The Crystal Tower storyline tells us that there were indeed survivors, G'raha Tia being the last descendant of royal blood;
You see, not all the world perished in Xande's calamity. Survivors stood amidst the ruins of Allag, looking to the Crystal Tower. But it was nowhere to be seen.
Yet they hoped its spires would again dominate the land, and prayed that the Crystal Tower would be a beacon of hope to people everywhere.
Word of this soon reached the only member of Allag's royalty to outlive the empire: the princess Salina.
Salina was moved by the people's dearest wish. She used the very best of Allag's technology to give her blood and memories to he whom she trusted most. That man was my forebearer.
For millennia, we waited. Allag's art and ingenuity faded. Our royal blood grew thin. But before the last drop could vanish from the realm, the memory of all this returned to me.
Rather than being wiped out it seems more like their races identity just slowly died out following the Calamity without their empire holding them together.
I'm honestly more interested in their society. All we've seen so far is basically their military might, which was mostly built around subduing Primals and conquering the world. Dialog suggests they had a civil war on their hands as well, with Allagan Warriors of Light being what actually caused Xandes to dump all of Dalamuds power onto Crystal Tower in an attempt to quicken his goals, but what instead caused the Calamity. They had royalty, I believe princess Salina was described as opposing Xandes and Amons world domination.
Silly theory ahead;
Given their technological prowess they seem like a secular society, and you certainly can't spot a single Twelve related symbol in any of their structures (I thought the statues surrounding the battle field for Phlegethon might have been something, but there weren't enough of them for them to be the Twelve), however the Boarderland Ruins does have a Sun Temple, which given how Crystal Tower powered their empire (before Dalamud) isn't too surprising, but oddly there isn't a single allusion to Azeyma (unless I missed it), it's also located close to a Markets and Manor, which suggests its a civilian area. This leads me to believe they either out right did not care for the Twelve (which given the possible similarities to Primals makes sense), or the Twelve simply did not exist at the time. However, we do have a collection of Allagan Runestones dotted about in various Levequests, featuring the symbols of the Twelve. These are however, very basic even by modern standards, we farm Allagan Tomestones frequently which seem much more complex, but these are literally just carved stones. This suggests to me that they either originate early in the Allagan Empire, or after its fall. I'm going with the later because of the Allagan Runestone - Menphina; it features the twin moon (Dalamud) on it, something they put there in the twilight of their reign... Now, to me this strongly suggests that, following Xandes Calamity, the survivors created the Twelve mythology. Could they have simply have cast aside the Twelve long ago and the Calamity renewed that faith? Maybe, but if that were the case I'd expect the Allagan Sun Temple to still have some allusion to Azeyma, and it doesn't.
Now, my crazy theory is this; The Twelve are/were notable Allagan survivors. I always found it odd that the Twelve have relationships, for example Rhalgr is described as being father to both Byregot and Halone, yet their mother is not mentioned. Right off the bat that kind of lore makes the Twelve seem less divine and more mundane, and if the Twelve were Allagan survivors, you can explain the absence of various mothers/fathers; They didn't survive the Calamity. The Twelve who survived the Calamity and started leading survivors through the Umbral Era became legendary amongst the survivors, eventually resulting in them being near worshiped by descendants to the point where whole city-states do end up worshiping them as patron deities, in much the same way that we can perhaps view Bahamut as having once existed.