They could easily fix this by tagging specific worlds as "RP Worlds" so that people that are into that kind of things know where to go, and then add an option in Duty Finder that you can check/uncheck wether you want to queue with people from RP Worlds or not (like the option for queueing with people that speak French, Japanese or German)
TERA used to do that back when it had more than 1 server and it was widely used because people from RP servers are generally bad at video games and other people didn't want to queue with them, specially for PvP.