I personally believe you should be able to craft with items currently on your retainer, you could even make it that you have to be a certain distance from a summoning bell.

in its current state with the amount of materials in the game you have to keep track of what retainer has what items and its very easy to have different stacks on multiple retainers

having access to these materials from the crafting menu would relieve some of this stress or you could just bundle all retainers inventories into one

One last thing having a "Search" option on retainers would be nice

thanks for your time