Who's to say garleans couldn't summon primals? they may not be able to cast spells but nowhere does it mention that ability is needed to summon a primal. Besides the empire could be 'good' by general standards under different leadership, I doubt all garleans think the same way and Varis can't be emperor forever. (unless they wanna go all warhammer 40K on us)