I think it would be a great idea to allow you to visit other worlds and have access to your retainer. It's already annoying to disband a party when you travel to a different world. Then restart the party invite.