The system does get abused a lot, because there is generally no risk involved. There have been many times I've been in a dungeon and the "Vote to Kick" popup shows to request my vote, for no legitimate reason that I could tell. So I simply cast my "No" vote and hope it ends there. Unfortunately, it usually doesn't, as the chatwar then begins.

Perhaps if when a person is kicked, the DF doesn't begin searching for a replacement for 5 mintues would be a deterrent. People might think twice about kicking if the party member can't be replaced quickly.