Not sure if it's related or not, but I have a similar issue where ffxiv starts lagging increasingly worse to the point of being unplayable whenever I unplug my headset. Not only does the audio disappear, but it's like the entire game just falls flat on its face. I never understood why, and it has left me completely dumbstruck.

If it's related, any insight would be awesome.