Whoever was shouting was being stupid. I do just fine in hamlets as a fisher, and in fact it's the only class I have that's useful in there. In hamlets, a gathering class is a gathering class. Just be sure you have you Gathering stat above 80 points. It's literally all you need.