Let me put this in context because for some reason you seem to think just because the Garleans think its fair makes it so. First in comparison to Duskwights in Gridania those who are in Annexed territory are in fact conquered. Now and I quote ".. indicates that they lack even the most basic rights afforded to a citizen" to put that more clearly they are slaves. The "tough but fair" method requires 20 years of slavery and while you get to be treated as the lowest of actual citizens after this there is still a barrier between the Ban/Cen and the rest. While I was wrong about the theory(though really in practice its unlikely works out). The Garleans are still clearly..not a meritocracy.
As for the racism endemic to Gridania for the Duskwights and the Keepers we see in the Archer and Lancer quests while YES they do face racism on par with real world racism they aren't legally hobbled in the same fashion as Garlean Aans. The Laws still apply to them even if a Duskwight might get arrested for 'walking while duskwight" that's not a part of the Gridania legal code, while the lack of rights an Aan has in the Garlean empire would mean a true Garlean can just casually kill them and answer to no one(See Nero in his introductory scene).
Rhit is the only higher up Imperial we see who is not a True Garlean and I will once more point out he is the lowest ranking of Gaius' elite. Not a single true Garlean serves him. The fact that he's a slave in charge of other slaves doesn't change the situation nor the fact that the power granted him by such a position does mean that his commentary about why he serves is suspect.