According to the original lore book it's also worth noting that Northern Ilsabard is practically inhospitable. If not for the presence of Ceruleum, the Garleans would have died out. Add in their lack of ability to manipulate Aether bar a few rare exceptions and they endured generations of a lifestyle that wasn't particularly pleasant. Magitek is a rather recent discovery as is the invasion of areas outside of Ilsabard - and as we now know, that can largely be attributed to the Ascians.

I'd consider the Garleans to be reasonably justified in taking control of Ilsabard itself whereas their claim to anything beyond their borders is not justified at all. As always, my hope is that Garlemald will return to a Republic yet maintain control of Ilsabard so that it can stand as an equal with the likes of Eorzea and Hingashi. If the Garleans control Southern Ilsabard, they'd be able to grow their own food more effectively as the climate is more agreeable. If they lose control of it, they'll be forced to rely on foreign trade which is a recipe for potential disaster. After all, ceruleum is finite and they require it to survive the harsh environment that is Nothern Ilsabard.