Can't you just right click off the buff if it annoys you so much? Or make a macro for it?