I have 8 retainers.

It's not so much for storage (though I do store different things on different retainers) as most of my general HW crafting stuff all fits on one retainer.

Rather, it's for Ventures. It's so I can go off and run dungeons or craft other stuff while they're out getting me materials that I don't want to have to pay Gil for. I have two Mining retainers. I have two Botanist retainers. I have three combat retainers, and one Fishing retainer. That breakdown is what seems to me to be the optimal breakdown in terms of materials gathering.