Listening position has a big impact but it's hard to prevent it from getting louder sometimes because either your camera or your character will be near them eventually. You can try to keep your camera distant though.