It depends on whether you are dedicated to crafting and gathering, where it might be convenient to send them off to collect materials at the end of the day.

Don't expect to make a lot of gil using your retainers to gather things, as you and everyone you don't know in the game are already gathering stuff outside of retainers.

Use them for storing items you want to keep around. Two combat retainers is fine. If you have real doubts, and you also took up botany, make them miner, botanist, and combat-of-choice. Note that they can never outlevel you in a particular job or class.