It's those random virtual device reconnects caused by the GamePass system that causes the hickups, you can check the Windows Event Log, there are entries every time you get a controller timeout.
I've currently removed all GP games and also disabled the polling in the registry via this change:
Not perfect, but works (GamePass is overrated anyway)Disabling "Microsoft Device Association Root Enumerator" under "Software devices" fixed the problem:
Device Manager > Software Devices > Microsoft Device Association Root Enumerator