It's possible that first timers get queue priority which might be why you get almost insta queue the first time you do anything usually. If you don't have first timer you might have to wait for everyone to beat it (the ones with the first timer flag) once before queue times normalize. Have experienced first couple days forever queue after clearing it on healer, too.