Are your hotbars locked? If you have the main hotbar (hotbar 1) enabled, there'll be a small lock next to it that'll either fix abilities in place or allow them to be removed/dragged around. If it's not enabled, should be as easy as popping into HUD Layout, activating it, modifying abilities, and then deactivating it again if you don't want it there.

(At least, this is how it is on PC. I don't know if PS4 is the same way.)