The dwindling community is the problem. If we had enough people, all the lower rank people could match with people their own rank, and all the higher rank people could match exclusively with higher rank people, and the queue times would always be under 5 mins.

More rewards needed. Not just top 100 ranked, but something attainable for everyone else to make them want to play. Play X amount of matches in a season to get something nice. Every season could have both gearsets and mounts, and just rotate around which reward is assigned to playing lots of matches and which one is for being in top 100 rankings.