For the past several centuries, the Duskwight Elezen have lived in the woodland caverns of Eorzea. These cave-dwelling Elezen are the descendants of a branch that split from the main Wildwood clan during the founding of Gridania. The Duskwight despise the "shackles" of urban life, and it is not uncommon for this reclusive people to avoid the city-states altogether.
From finalfantasyxiv.com's description.

Just seems they don't like structured society.