I think a good countermeasure to ensure that people aren't likely to kick people just for turds and giggles would be to simply have a minimum amount of time that must elapse, say, 10 minutes, before vote kicking is possible. This would give the player the chance to demonstrate to the others that he isn't going to be a problem before they immediately oust him.