IT ADDS IMMERSION BOO...

In all seriousness, I've only had this happen to me every so often. Yeah it gets annoying, but if that was the case might as well take the Silence spells out, OH those annoying poison spells too God I hate casting Poisona.

Really though, your focusing your attention and concentration on casting a spell. If you are being attacked, something is wrong unless your are soloing... which XIV doesn't cater to soloers as much anymore. Find a group, and you won't have this problem