The narration says that Eorzea is one of five continents. It is made clear in 1.0 and again in 2.0 that the adventurers are not from Eorzea. At this time we know nothing of the other continents except the empire, which it also is clear that they are not Eorzean either. It is possible the adventurers are from those other places, and also it is possible the empire has already assimilated the other continents. Perhaps that is why the adventurers don't mind fighting against the empire, and perhaps are more like refugees. However, in my own back story from 1.0, my character is from the same continent but from an area not occupied by any of the city states.