A vote-kick will require all 3 other members in a 4 man, and probably at least 5 in an 8 man party. There's nothing wrong with that. The damage done by trolls and afkers is far worse than not having vote-kick.

Simple solution if you don't want to be afraid of being kicked: Don't go into something undergeared unless you're doing so with friends. Though this won't be a problem when they add gear requirements in 2.1 (thank goodness).