This has been happening ever since Endwalker launched. As far as I know, this only happens with this game. I've tried this with my wireless Xbox controller and my (brand new) wired one as well. Basically at random moments, the game will stop receiving all inputs for a second or two. My controllers do not appear to be disconnected when it happens since the guide buttons aren't blinking. The drivers are also up to date. I have tried two things so far:
I've tried disabling Microsoft Device Association Root Enumerator, and I also tried disabling a certain thing in device manager that has to do with Xbox Game Pass(more info in this thread on reddit )
Allegedly, i'm not the only one with this issue, but I'm hoping something more can be done as this has gotten me killed in various moments such as savage raiding or random dungeons.