Here's the thing though, you're assuming that you can just continuously spam skills and you can't, and this is where crit jumps ahead.

Fights like Titan for example, demand you stay mobile and thus interrupts your attack cycle. In these instances, skill speed is wasted because you can only nudge in so many attacks before having to move again whereas Crit Rate can greatly increase the value of said limited strikes. In a vacuum with unlimited TP, skill speed would likely (eventually) win out with enough of it stacked. However, from what I've personally seen and read (does anyone know if Kanican still plays?) the consensus does appear to be that skill speed is trash. Spell speed is only marginally better because it also affects casting times and not just the recast which can be a big deal for WHM & SCH.

Given the choice, I'd take crit rate & determination every time over skill speed.