Yeah, the weapon upgrade is unquestionably the better choice in this case regardless of secondaries.

If you really value spell speed that much, the Dreadwyrm Cane is (unfortunately, IMO) loaded with it.