Would it be broken to just do away with spell/skill speed and just replace it with "speed"?

Currently jobs are pretty much split into physical and magical with very little overlap anyway, the only real exceptions to this are pld and drk and in these two cases the lack of ability to lower the cast time/GCD on the spells is something a lot of people complain about (looking at you clemency).

I just don't see the point in having these separate stats that could easily be made into one thing and wouldn't have much of an effect on game balance while also being a slight positive to jobs that have spells and very limited access to spell speed.