I am going to argue just a little bit, because I have two MIN retainers. That is how heavy its demand is, and this demand is present regardless of what your crafting goals are unless you deliberately avoid 80% of crafting requirements. The demand for metals by themselves easily totals, if not exceeds the demand for all other mats combined. While classes such as CUL and ALC are BTN heavy, this is because of the amount of different mats needed to craft those goods. This demand comes nowhere near the sheer quantity requirements for ores, stone, and sands.
The thing about retainers, is you need to send them out in advance. If you're all of a sudden hit with the need to make 200 ingots, retainers taking 40-60 minutes to bring back 30-50 ores at a time is only going to extend the amount of time it takes to complete your current tasks since you can gather what you need in much less time. Your retainers should be used to help fund your needs. Stock piling mats is a distant second.
Mob mats are only more painful because there isn't a gathering class that specializes in their acquisition. Combat retainers simply take too long and the their yield is too low for them to be of a benefit. Again, you will gather much more on your own in the same amount of time, despite the RNG factor with mob drops. The only reason to have a combat retainer is for things like quick and standard explorations as there are items they bring back that are exclusive to those type of retainers. Or you can use them to bring back the mob mats, and then turn around and sell them since the demand will always be high for them. In later levels you can use the stones dropped from FATEs to buy mob mats. Again though I feel this is a waste, as those gemstones can be used to acquire exclusive items such as orchestrion scrolls and vouchers. Don't burn up those stones on mats man.