I figured out the issue with mine. The game (and my PC) was recognizing my keyboard as a controller. The game would intermittently prioritize the keyboard over my XBOX One controller.

I went under Devices and Printers > Properties for the keyboard and disabled the "controller support" function. My actual controller works fine now.