As others have mentioned the time you're trying to get your PvP done is not a popular awake time for the folks who PvP.

However, are you queuing up as a freelancer or with your Grand Company beause of your not queuing up as a freelancer you're going to have longer queues for PvP in general.