I guarantee you won't notice anything by missing out on 7 points of spell speed. From what I understand, you need a LOT of spell speed to even notice it.

Plus, if you have any plans of doing SCOB or FCOB, you will need accuracy.