Make it an option that can only be done by talking to the Retainer NPC, the one you talk to when you are first getting your Retainers.

This makes it a very deliberate action and not something that can be done accidentally.