Didn't know others had this problem.

I always have to relog if I want to plug my headset indeed (if I want to go on discord for example).
When worlds were congested, I avoided doing it because i would have been in the queue again xD.