This seems tied to how you can't join a party if you are engaging your retainers through the summoning bell. Seems like an underlying code decision made a long time ago by SE - a restriction on what you can do when interacting with NPCs. It is like crafting has you interacting with a crafting table which might as well be an NPC.

It would be a great QoL change though.