The buff goes away either once its time runs out or you gain a certain amount of XP, I suspect what's happening here is the latter. I was also very confused back when I was grinding gatherers in Diadem!