I personally have 4 Miners, 4 Botanists, and 2 Combat. I had a fisher originally but converted them to BTN and MIN. My reasoning is that fishers can't bring me back crystals and other useful mats in the long run. You get more out of the other retainers and if the fisher gets an item locked behind it, I should be able to afford it over time as it'll get cheaper. Even then, combat isn't as needed either but still useful for the random hides and stuff you might need for crafting.