I pay for all six extra.

3 combat, 2 miner, 2 botany, 1 fisher. The primary reason is for materials acquisition. I prefer to have some materials on hand and then send the retainers our to restock while I"m doing other stuff.