Blame the requirement to only be able to queue for a specific GC, and having your PVP rank tied to that GC, above all else. I guarantee you at most times of the day there's at least 48-72 people across an entire server willing to start a match, but the reason for the wait is to fill up the ranks for all three GC's and start a match.

Guessing you're prob not in Adders... during prime time I get impatient at 15 min mark.