It's only really infuriating for things like silences or stuns.

Things that really need to be used reactively.

So you end up doing nothing for 5-10 seconds while you wait for the ability you have to interrupt, instead of approaching combat like one would expect.

To call them "off GCD" is entirely misleading.