It's actually a really easy solution. Add in game parser. It allows you to see anyone you grouped up with, you can't see random people from duty finder. That's it. And for those who will say it will cause kicks from party finder, the groups who will kick are already parsing.

BTW there is a way to kick someone from PF with out actually kicking. Whoever is pulling there weight whispers or gets whispered, "We are desbanding the group to drop X player/s, we will reinvite". It happens all the time.