Uninstalling the Game Pass games seems to have fixed it for me. I've been running Eden Savage the last couple hours without any problems.

The exact error I was getting before in Event Viewer > Windows Logs > System:
Disk 4(-9) has been surprise removed.

Maybe these do a check for updates and have to remount the Xvd devices, causing the Device Manager to refresh.