It's a known 'feature'
>with multiple devices
which is also how you deal with it, by only using 1 device while xiv is open.
as long as the device itself doesn't change, you won't have to restart xiv. I can swap my speakers/headphones freely if I unplug/replug them into the same sound card jack as the device itself still maintains the same state. It being default has no bearing - it's the act of switching devices that xiv seems to hate. You could also use an external DAC or a similar hub so that you can switch audio without switching device. I've heard people to have success with Voicemeter as well, as a software solution, but haven't tried it myself.