I ended up buying the 2 additional retainers because I was running out of inventory space for all my crafting mats but here is what I do:

While leveling gathering, whenever I come across a new item I look it up on http://ffxiv.gamerescape.com and check what recipes use it. I actually even go as far as counting how many of the item I'll need to make everything (I'm a completionist) and gather at least that many. Then every so often I'll go through my retainers' inventories and check each item to see if I still need them at my crafting levels. If not, I vendor/market them.