I'm for a vote kick, but it needs conditions.

1) In a light party, must be 3/4 vote. Full party is 6/8.
2) Vote kicks can not be initiated while anyone has enmity or is within a CS. If a fight starts while the prompt is active, the kick is canceled.
3) Vote kicks can not be initiated if there is treasure to be distributed.
4) Vote kicks can not be initiated until 3m after a boss kill/wipe has passed if a boss has been engaged. (Giving more than enough time to trigger #3's condition)
5) You may only initiate one kick attempt per instance, refunded if the attempt is interrupted by #2.
6) Failure to vote within the time of the prompt will be considered a No.

...it's not perfect, but ultimately it should minimize griefing potential.