There should absolutely be an option to "Remove from Residence" if you right-click any player character model in your own home, as well as tools to keep that player out while leaving the place open to people you want visiting you.