I've noticed this as well inside my private chambers within my Free Company's house - the problem seems to occur when in the back of the room close to the windows (and it's especially apparent during night time). I have an orchestrion in the room playing music and yet the outside ambient sound effects are still apparent when close to the back wall and windows of the room (amd yes when it's raining it becomes even more noticeable).

It's not a game-breaking fault and it's something I've been mostly ignoring, but seeing this thread made me realize that I'm not the only one who has noticed this.