The Gridanian attitude towards outsiders is one that several Gridanians themselves comment on as an issue. You can see how it is rooted though. Remember it took hundreds of years of having to live underground before the elementals finally accepted the Gridanians and allowed them to live on the surface. This makes the Gridanians rather paranoid about maintaining their relations with the elementals.

As for why so many groups evade retribution, I'd say its probably linked to the fact the elementals were weakened by the Calamity. I also suspect the elementals allow the Gridanians to take responsibility for much of it. If you have played through the White Mage story you would know the forest was getting pretty angry and needed placating.