You don't need to stay for the rest of the quest. Once they accept it, you can leave. On the last one with 4 people, I messed up trying to go around the Spriggans in the end. Everyone else did theirs on the first try and left. I was able to continue the quest without them, so you just need them to trigger the quest.