I wouldn't say barely, high SS vs low SS can be the difference between being able to do certain openers or not, or being able to get a cast off before dodging or not. The speed is also very noticeable on MNK, where high SS can get your TP dry but low SS doesn't.
Crit and Det just increase damage, but speed can affect your gameplay either positively or negatively, some classes feel very smooth with high SS.
