I definitely agree there's use to it in endgame.

However, early in the game, when you get stuff that grants like "Skill Speed +3"... that's something that should definitely be replaced, because you can't even start stacking close to enough for it to be meaningful.