I saw a Reddit post not too long ago that someone wrote that they lost their house because although they spent a lot of time tending their garden, they didn't actually go inside all that much, and as a result, the demolition timer expired. It surprised me that interacting with your garden wouldn't reset it.