This would conflict with everything you do with the D-pad and the four action buttons that isn't using a skill off the cross hotbar. With this scheme, you couldn't switch hotbars then jump, switch hotbars then change target, switch hotbars then cancel cast/target/whatever, etc.
Not a problem if it's an option you can turn on and off, but I personally wouldn't want this without an off switch.