Pretty sure this is because the cast bar isn't just an animation – the progress bar actually validates against the server, and as we all know, the client/server asynchronism in this game is unjustifiably bad.

If it's any consolation, that cast bar has nothing to do with your actual casts anyway. The best example of this is when summoning your chocobo / mount. The cast bar finishes, you move, and the cast cancels. The actual "cast" is not registered until the whistling animation completes, which happens after the cast bar fills.

In other words, don't pay much attention to the cast bar. It's the animations that matter in this game. For better or worse.