X-server grouping is not the problem. Players being a douche is the problem, and that's going to happen whether you have X-server grouping or not. So we might as well go with it so we can have the convenience. Besides, if the thought of using it scares you so much, here's a thought:

Don't use it.

That way, you won't be bothered with it, and everyone who wants it can still have it. Everyone wins!