People with the "that's selfish!" argument are always so weird to me. Most behavior in games is selfish to a fairly high degree and there's nothing wrong with that. If you're not having a good time, and leaving the party would make you feel better, then feel free to leave. Nothing involved is important enough that it's a big deal. Most activities in the world understand and support this because they understand the person's well being trumps some odd notion of whether it's selfish or not.

"It may affect other people negatively" is nowhere near enough reason to avoid doing something.