Counterpoint: You stole the a house, a limited commodity by taking it and then refusing to do anything with it thereby denying it to someone who might have actually used it.

Its not a unique thing either, there are plenty of medium and large houses out there that have been empty aside from (maybe if you're lucky) a single merchant in them. I'd hate to think there are players out there who are just so petty they'd keep a house and do nothing just to stop others from actually enjoying them.