No, no, you've got a point. They only take away religious freedom. Well, that and they don't offer full citizenship without 20 years of military service, will forcefully conscript conquered peoples into said military, and then remorselessly kill them for slight infractions. There's also Varis' intent to genocide the beast tribes instead of trying to find a diplomatic or at least peaceable solution to the Primal issue. You're right, though - the tales of their cruelty are widely exaggerated.
/sarcasm
Anyway, while neither the Roman nor British Empires were strictly "evil" or without merit, they still weren't liked by the majority of the populace (esp. in conquered / claimed / annexed / colonial territories) because they were oppressive just as Garlemald is. People don't like being oppressed. Given Ala Mhigo has been under Garlean rule for... what, 25 years? Even if we do manage to liberate it I wouldn't expect it to go back to Ala Mhigan culture right away or... ever fully, possibly.