Its part of the greater issue of stamina.

There are two mechanics (cool downs and stamina) that should be their own systems, not merged into one.

It becomes increasingly obvious when using self buffs that consume stamina (and time)