The main reason why the queue times are huge for pvp is that not many people actually want to do it.

Why? Well, I'm someone that did tons of PvP in WoW, SWTOR, and lots of MOBA stuff. I rather enjoy PvP. However, the PvP in this game is limited to a 4v4 arena deathmatch mode and has little real point to it. That's why I choose not to even bother in this game.