From what I understand, each class has it's own fatigue, so if you hit fatigue for one you can gather for another class just fine. I think it recovers fairly fast, don't know exactly how fast.