Again, they're not evil. Antagonistic doesn't equal evil. Yes, they attempted to invade Eorzea but they've never thrown their full might behind doing so - Nael and Gaius were essentially acting on their own and whilst Varis is currently within Eorzea he seems more concerned with exploring Azyz Lla than invading Eorzea.

Azyz Lla isn't really Eorzean territory. Yes, it's technically in Eorzea but nobody had claimed it as their own until Garleans and Eorzeans clashed there. Again, they're very similar to the Roman Empire - and the Roman Empire wasn't evil. Morally dubious, yes - but not evil.