To close your retainer interface, you should just have to walk away from the summoning bell, not back out of every menu one level at a time.

Walking away works for the market board and vendors, so why not retainers?