Seems to me that random, non-party visitors shouldn't be able to change anything in a house, let alone make it disappear.
Now, on the other hand, it might be interesting to have a toggle that allows visitors to move furnishings any way they want.