Ya, time to do something about them.
Today I was logging in before primetime in order do the event, getting in is not a problem around that time.
Then to enter my house in order to deal with retainers which I like to do inside my house. Took a few tries to get in due to the housing server being under a lot of stress.
Then I did the event and wanted to get into my house again, this time I was unable to due to the same housing server issue.

This is not on SE but rather on those who hide inside their houses while they do things in order to not get kicked for being afk, which they most certainly are. This is something SE should look into, perhaps by sending a big message that pops up in the center of the screen of those who stay inside their house for more then 1h, if the person does not respond to the message within 30min they get kicked. Pretty sure this would make it possible for more people to actually play.