There is nothing fair about being conquered. We also saw how the "recruit" in 1.0 and in the 2.0 quests involving the deserter. We can look at basic math to see that the majority of legionaires have to be conscripts because the majority of the Empire is annexed territory. The Empire austensibly controls two whole continents and were considered a small nation before they went out expanding. We know for a fact that the Empire took heavy losses in Othard fighting Primals to the point Primals now completely freak out the Imperials. Unless True Garleans are hyper fecund the majority of legionaires have to be conscripts.
Since we only see the Empire on war footing it might be a reasonable mistake to assume its a military meritocracy, but look again at the civilian hierarchy. Commoners cannot gain any of the political positions. Twenty years(or more so they could just decide to not give it to you.. but hey "tough but fair" right?) means you get the rights of a commoner. Your children(because they were probably born during your noncitizen time!) have to go thorugh it all over again and none of them have the option to have any say in governing.
Meanwhilte Uldah, which no one but the most Randian free marketist would call a meritocracy, has Raubahn's rising up from the streets to joining the the council that runs the city by raw act of his skill with a sword. That's far more meritocratic than anything we see with the Garleans but hey the Uldians don't scream about how fair and just the invisible hand of the free market is so I guess no one looks at that.