Keep in mind the elezen are both the sole native people, and originally nomadic in culture.
They may well of been a nomadic group unattached to the Gelmorra that merely settled in one place founded on a myth.