The empires biggest fault would be their pride. The rule of Ala mhigo for example wouldn't of been so bad if they actually offered any kind of assistance to integrate into their society but the garlean philosophy is built on a pride that originated from the disadvantages they had a long time ago. They had to work hard to get their technological edge with the odds against them so the garlean motto is basically earn your place 'just as we did'. The idea of savagery is born from the atheist mindset the garleans have 'we didn't need gods, only savages need them'.