As others have said the problem sounds like a typical bugg which has become alot more common after 1.19 and has nothing to do with affinity traits such as axemanship.

Cooldown timers will glitch and show shorter cooldowns then when the skills will actualy be ready to use again, which creates both confusion and frustration.

The only known solution is to completely restart the game, how it happends I don't know.
Sometimes it can happen in middle of a leve when I've playing for a few hours, other times it can happen just minutes after being logged on.