I can't really give you an honest answer as to what may be causing your specific problem, but 9/10 times either a restart of FFXIV or a restart of your computer will fix the problem with recast timers being off. I only seem to experience this myself when I AFK for 24+ hours, with the game left on and even then it isn't all the time.