when you're not inside the house it's effectively despawned. this is why you reappear outside. it only loads in for you when you enter and it goes away when you leave.