It really depends to your playstyle. If you prefer hunts and fates or anything combat-related that can let you gather crafting materials then your retainers can be gatherers. If you want to earn gil, you can either have your gatherer retainers farm crystals or combat retainers can farm crafting materials to sell to the marketboard.