Why doesn't skill speed work

[ElHeggunte]

Would there be something game-breaking to have skill speed bonus work this way?

Considering how powerful some cooldowns are, yes it would. Though right now Skill Speed has such a minor impact it still probably wouldn't be worth stacking.

"Cool down" and "recast" are two entirely different things.

You're arguing semantics. In this context "cool down" and "recast" are exactly the same thing, just different terminology. SE uses Recast, other games use Cool Down, both denote how often you can use the ability in question.