Ive noticed this special fatigue to kick in when I have been in one area harvesting for long periods of time. Not sure how the game calculates it, but I think when you hit a certain number of items in one area then you start to feel these effects. All I do is quit and just come back the next day.