But why should my group be forced to wait 5 minutes to get a new person after we got rid of someone who was causing my group grief? What is wrong with getting a new replacement as soon as possible? Heck, why should people that use the vote kick responsibly get punished because other people don't use it properly?

If people wanted you gone for the stupidest reasons, chances are they were going to be unpleasant to play with anyway.