Yes I consider hitting fatigue as soon as I break my first item ( I said : " I start to damage stuff" ).
I know I could continue, but I gather for the sake of looting items, thus breaking them, even if it is only 20% (which is a lot imo) just makes me stop.