I have the same issue, whether with headphones, speakers, or monitor speakers. But a user on Reddit found a partial solution for me.

Esc -> System Config -> Sound Settings -> Listening Position. I changed this from 10 to 100 as the user recommended, and the distortion drastically reduced. While it still pops and stutters for a couple seconds every few minutes, it is no longer near constant.

With a maxed out Listening Position, when it does stutter I can stop the (now short) stutter by moving the camera around. Then if I move the camera back to where it was the stutter comes back. So it seems to be something with the sound's position logic.