Imagine how the PS3/PS4 users feel who can't or would fumble to do that. However I currently do the same, right clicking the buff. For asts seems like it should be a key-bind option to cancel the draw, not so much a skill slot needed though.