Thats not entirely true if considering trade offs of even the worst stat det as the baseline rather than just no substat.
Also there are circumstances like with nin. Where having more than 700 skillspeed doesnt pheasably serve much purpose because we lose everything we would gain speed wise to mudra clipping. And anything over 1500 will misalign all our shadowfang dots from trick. Yeah occasionally we gain a gcd, but dependent on the phase, movement mechanic, or timeline of the boss we may not even gain that gcd and thus the misaligned SF is effectively a loss.
Additionally SkS does almost nothing for the first 30 seconds of a fight cause were dbl, triple (mudra), or 7 (TCJ) weaving for every gcd in those first 30 seconds.
Point being there are some classes where it does cause distinct rotational flaws and misalignments. Course thats more of a class mechabic issue than SkS, but i think you probably get my point.