Check the crystals in the cities on the hour, every hour, that is the time that inactive houses demolish themselves.

I know a lot of people who have done this successfully on Balmung, so you can do it too! I believe in you.