If they did it, they would need to put in restrictions. Possibly limited to housing/apartments and only if you have an item which you need to interact with in order to access retainer's items as eligible items to craft with. A potential downside being that while crafting in this manner you will not be able to sell any items on retainer.

You have to respect the way retainer information currently works. When you call a retainer, they can't sell items until you are done using them, so said restriction may have to take place in order for it to work. But that's the best suggestion I could give with the system. if it really is a hassle to go through your retainer's inventories to find the item needed and you aren't just asking for free range of where you can craft, I could see them doing it.

However, if you are asking for free range of retainer's inventory regardless of location, you are honestly asking for to much because you are suggesting a resource draining convenience because you can't be bothered to take the 1-2 mins it would take to just take it out of the retainer's inventory and put it into your inventory.

But that's just my take on it, they already have the resource to allow you to right click search for items, and they even tell you which part of the inventory they are in. Another possible suggestion I could offer is introducing a toggle option that will highlight items you right-click search for until you hover over them. So if you have issues with being blinded by the overwhelming amount of items in your inventory, I believe that could be easily fixed if they just introduced a way to provide a bolded border for items you right-clicked search for, possibly limiting it to up to 4 items.