There are a few inaccuracies in here that I feel obliged to point out.
It was at the foundation of Gridania, not Gelmorra, that the distinction between Duskwight and Wildwood came to be. With the invention of modern Conjury, the wrath of the elementals against the civilisation abated, allowing those who lived beneath a violent forest to emerge in peace. Those that emerged from the underground cities to live in the sunlight developed the more tan tones and associated features. The duskwight arose from those elezen who did not wish to leave their underground homes behind.
We do not know if the padjal existed prior to Gridania's foundation. There is no question that they emerged the leaders of the new nation as a consequence of their aetherial sensitivity and ability to communicate with the elementals, but what role they could have played in Gelmorran society is unknown.
It is possible that the true guardians of white magic are the elementals themselves rather than the padjal, and that the Seedseers are more often than not the elementals' chosen wardens (YOU being the obvious exception). The White Mage quests before the calamity refer to the "pact of Gelmorra", suggesting it outlines rules of peace and non-violence that must be abided by the Seedseers specifically, implying that they only hold white magic and their elemental sensitivity because of it. Of course, this could use some clearing up, as the information available has been vague at best.
We do not know if Gelmorra existed prior to the sixth umbral era of the great flood. My guess is that it did not, rather that it was what remained after the fall of Amdapor during that Sixth Umbral Era. If that umbral era was caused by the abuse of white magic by the Amdapori civilisation, you can bet the resident elementals of the Twelveswood would not have humoured their ilk in the centuries that followed, forcing survivors underground and claiming the remnant secrets of white magic under their own protection.
Many Gelmoran ruins bear padjali nomenclature, strongly indicating that padjal names are a remnant of Gelmorran traditions and the hyur civilisation of the day. In fact, many could have easily been re-purposed after their were no longer needed for residential purposes, transforming them into cellars, jails, crypts and the like. The ruins also bear the symbol of the entwined adders (but without the lotus flower), symbolic of the union between Hyur and Elezen - something that likely translated from Gelmorran society to Gridanian society.