If one of your retainers has the crafting material, you don't need to take it out from that retainer to actually use it in the item you're trying to craft.

This is the biggest reason why I never bother crafting or leveling up new crafters.