I have your perfect fix:

Stop rendering the retainers and switch to a menu/text based interface. You can keep your player-owned shops without needing to have a physical shop.