If you automatically equate invasion with genocide then the Garleans are genocidal monsters that need to be put to the sword because of their invasion, subjugation, and enslavement of the nations surrounding their homeland and did not even stop there, working to seize the entire continent they are on and when the going got tough because of the primal threat their response was to drop a moon on it knowing full well it would result in the complete annihilation of all life in eorzea. With the only people against it wanting to conquer Eorzea so they could ship its inhabitants across the sea in every direction to be used as frontline fodder in further invasions across the globe on any undiscovered landmasses in their goal of taking over the entire world for just pureblood garleans to live in.
Us doing a retaliatory invasion of them is just standard politics. If you stretch it a little you can find some parallels to World War 2 in the current state of the story. Not to mention that Garleans who are peaceful are generally accepted by Eorzeans. So us going in and being all genocidal is purely locked up in your head because you want an excuse to ditch the people who have only been fighting defensively to reclaim their own land and the land of their allies.