I had controller problems for a long time which was really hard to pinpoint the cause. I think it turned out to be something silly like my computer didn't like the usb cable my controller was connecting through. One of my different cables has no problem, and wireless mode also has no problem.

I'm guessing the issue you are describing is randomly losing connection while playing, causing your character to be stuck moving uncontrollably in the direction you were last pressing. Wireless mode on both XBO and PS4 controllers works well for me without those issues.