Simple:

Vote Kick.

You just have to make your case. If they're AFK say so before you submit it, and use the AFK option.

If the other party doesn't agree with you, that's the way it goes.