There was a concession made for it in the MSQ when leaving Eulmore after meeting Vauthry for the first time:

Duliachai - "And I shall make arrangements so that you can pass through the gate without causing a stir. I expect you to come and visit."

So it's thanks to the Chai's that you can enter whenever you want.