The guards are very clear that any time you are permitted into the private chambers of the leaders, it's because the leader invited you in and they have been told to expect you.

Basically, access is granted by appointment only.