The delay seems directly tied to the same delay in being able to use a GCD ability.

It seems that the same amount of time where you have to wait from your first step of a combo is the same amount of time you can't tap the right bumper (RB) to swap cross hotbar sets. If you wait until your GCD timer is half done, the RB responds immediately, so I'm guessing that the GCD cooldown and the RB tap to switch has somehow been linked. Using the RB + x/y/a/b seems to remain unaffected.

Like others have said - this is a really big issue for people that have gotten used to tapping RB mid-combo to use non GCD abilities & buffs, then tapping back to the combo.