I'm not over-clocking, but my CPU does seem to be a bottleneck, IE my graphics card is possibly a bit too good for my CPU. I run a lot of applications in the background too and many FireFox tabs because I am a web developer. Probably over-stretching the CPU. I wonder if this could be the cause.

I do browse the net a lot when idling or crafting but that's the whole point of windowed mode. I've found switching to another window while changing zones is a bad idea so I generally avoid that. It's probably windowed mode only issue. Yoohre_WildRiver's suggestion is worth looking into more. I think if we can pinpoint the cause we could post a better bug report rather than saying "this sometimes happens, you may not be able to repeat it yourself".

As for being logged in a while, it definitely makes it worse. I've had timers off by over a minute before. But as I say in my OP reloading the game in Ul'dah last night and then teleporting to Bald Knoll was enough to throw them out by about 5 seconds.

I don't know why they can't just handle timers better. Sloppy coding I guess, much like everything else.