I always had issues with my XBox One Controller Wireless. The game never recognized it properly at launch and I would have to change it in the menu like you described. It worked fine if I had it connected via USB, though.

I've since changed to a PS5 Controller and have never had a single problem.