My head canon is that the Garleans are descended from the remnents of the Allaghans who were cursed somehow to unable to use magic following the fall of the Allaghan empire.