None of my queues are over 5 minutes, except PvP queue.

Play jobs that get fast queues if you want fast queues. If you want to continue playing a role that is (falsely) perceived to have zero responsibility (fyi this is why people are still stuck on T5/T9, and why NA servers lag sorely behind JP in T13 clears), keep playing dps.