The queue time depends on how many people are leaving the game (or most likely being disconnected). The more people leave the more people can get in.