It's probably something to do with right now if you remove someone from your friends list it doesn't remove you from theirs. They have no way of knowing they're no longer on your friends.

If they were to add this in, people would know when they suddenly lost access to the house, but others could still enter etc.