But it should require a majority of the party, not just a majority of the remaining people besides yourself. Consider a 4-man dungeon in which two players want to do a full run and two want to race ahead and skip everything they can. The majority of the party has vote-kick authority, but which side is the "majority" when it's two against two? As it currently stands, it's simply whoever clicks the "Kick" button first who therefore qualifies as a majority even though half the party disagrees with them. You shouldn't be able to act as a majority and kick whoever disagrees with you if you're not in fact in the majority. In a two against two situation, neither side should be able to make unilateral decisions like that.
