To be honest, you answered your own question when it comes to the Empire and why we're not rushing to sign up.

They are FORCING change on people who don't want to change. They don't just want to get rid of the primals. They want to get rid of the Twelve, who like the Primals, they consider false gods. They indocternate the children of any country they take over so within one generation they are Garleans and have forgotten their own culture.

Look back in our own history and you will find many instances of the same thing. From the Spanish Inquisition to what the Europeans did to the Native Americans, to the purposeful destruction of the Cajun French language in the 50's. One side believed they were 'saving' the other side. While the side being 'saved' thought of them as villains trying to change a way of life that had served them well.