From what I've read, they first encountered Primals in Doma, right? And it's Primals that made that war as nasty as it was, so that's probably where the fanaticism against them specifically comes from.
The story doesn't really deny that that's a reasonable reaction - we think the Primals are bad too, after all. It's really more...everything else about Garlemald. Nations going on a campaign of brutal conquest for its own sake aren't exactly considered the good guys anymore, ever since we realized what a bad thing colonialism was. And, as I think has been mentioned, it's not like they're even letting the conquered people reap the fruits of their advanced society - every single non-Garlean will be a second-class citizen at BEST, and very likely little more than a brutalized conscript (if the methods used on the Ala Mhigans are any indication). For every noble goal of theirs, they have two things about them that make them undesirable as conquerors. There's very little about them that would make them seen as benevolent in the eyes of the conquered.
EDIT: To actually answer your question: I think it's abundantly clear that they're conquering Hydaelyn because they can and because it makes them feel Important and Strong. It started with their own land, it continued in Doma, and they turned to the last piece when they were able to after that went pear-shaped.
Your implication seems to be that something triggered their massive campaign against the rest of the world, but there doesn't seem to be any indication of that. It couldn't have been the Primals; the Domans' summoning of them caught them by surprise. Sure, it then informed their approach to Eorzea (Silvertear was because they wanted to avoid another Doma by pre-emptively stopping the locals from summoning Primals in reaction to their assault, remember), but it wasn't the raison-de-guerre. I really think it's just good old-fashioned, self-important Imperialism at play.