You should be able to just click on it and remove it, like food effects. At least, you could if you received the buff during the main story quests.