Maybe we can have a setting:
(x) Display my iLevel (DPS/HPS)

And then we can filter to only allow people who display their ilevel to join the party? Provided their iLevel is of the right level of course.

This way party leaders can create parties where only those who display and are of suitable iLevel join.
And people who are not interested in iLevel can easily avoid those parties.

This of course has the nice side-effect that it makes it an easy way for people to choose what kind of people they want to party with (those who care about numbers and those who do not).