It does matter, because it sounded like in the example you gave that the rest of the group communicated that they wanted to figure it out themselves. You're the entitled one there thinking that only your opinion on the subject mattered.
Nobody faults you for leaving. It's this attitude you have that *they* were the problem because of what they wanted instead of you just not jiving with that group. They might even have kicked you to avoid the penalty if you had asked. I've certainly been happy to do that if someone doesn't want to continue trying.