With 4 people, and only 3 voting, it doesn't make sense to start disqualifying folks. Are you saying that if a pre-made party of 3 goes in, they can NEVER kick out whoever joins them? Even if they're AFK, Offline, or spouting off every kind of hate speech imaginable?

Leave the system as it is. If someone's abusing it, report them to the GMs. That's the sort of thing they can (and WILL) actually deal with.