I always just accepted the next quest (force of habit so I don't lose where I'm supposed to get it from), and then went into the inn, so maybe I just simply never noticed...