There is a road in the Sylphlands that leads to Ala Mhigo, so we can assume at some point in the story we have to retake it back from Garlean control. At that point we will be on the doorstep of the empire.