I believe you can click the buff off, so you don't even have to use it if you do not want.