Correct me if I'm wrong, but in 1.0 Didn't Nial van Darnus regularly rant about being a direct descendant of Allagans? If so, I have a bit of a theory, the Garleans of present day were born from the survivors of the Allag Empire. It would explain why the Garleans kept being driven out of every place they settled. After all, the White and Black mages went through the same thing during/after the calamity they brought about. Even to this day White magicks and Black magicks are considered illegal (except in certain circumstances in Gridania).
And as we've learned from both the RDM class quests, AND in Shadowbringers, aether goes absolutely mental, an uncontrolled force, unbalanced, would that not have an effect on people near the center? After all, Aether is exceedingly connected to life itself. Wouldn't it becoming unbalanced cause mutations in people? Mutations perhaps like severing the ability to manipulate aether. Or a third eye? And what few Allagan survivors likely would have been fairly close to the Crystal tower that had become the center of their empire, thus closer than other races to the epicenter of the Calamity.