It would probably be fine. Maybe someone would whine at you that it's not "optimal". You can tell them to stuff it because things will really be OK with fairly small differences.
RDM doesn't have anything that would cause SpellSpeed to give outsized gains. But really they don't have "extra" synergy with any of the secondaries (that is like crit procs, BLM's particular phase style, tanks lack of direct hit w/o melding).
It really comes down to how much worse spellspeed is than the others at increasing DPS in general. The answer has usually been "it's not that much worse but you drain resources faster".