SE really needs to address this issue. There's a reason every single game that I know of, with housing, has some manner of controlling who can get in and you can't on an individual level.