I absolutely hated it when I had just hit 60 on my first class (BLM). I still didn't like it when I had just hit 60 on my second class (AST).
Since gearing up those 2 classes and also running it on non-caster classes for the first time, my opinion changed; I only consider it a mild annoyance now. Still, I'm 100% happier when Fractal pops up, but whatever.

On AST I have no trouble with the final boss at all.
With BLM, it's still...urgh. Not the adds, not even the whirlwinds. The worst part is that it ALWAYS goes into invincible mode whenever I'm on my last 3 seconds of Enochian and in the middle of my B4 cast -.-
In the rare case that it isn't, I'm surely getting hit by that abnormally large hitbox while trying to target one of the adds, which will make me auto-target the invincible boss and make me throw my next spell on him, canceling out any astral fire or umbral ice stacks I might have had left. And by the time I change target and finally get those stacks back, Enochian will have run out. Same with the first boss, btw.
Guess I just generally hate bosses that become untargetable for a certain amount of time as BLM...