The in-game mouseover for Spell Speed says that it affects both the casting AND recast timers for spells.