Xbox One controller for Windows over bluetooth not working

As the subject says, I have an XBox One controller for windows, and it is one of the newer controllers that can connect over bluetooth.

The problem is, over bluetooth almost all the buttons don't work and the screen spins as though the controller is pushed. I've tried running the calibration in FFXIV but it doesn't work, the same buttons don't do anything... so the game just isn't recognizing them.

Weird part is, if I use the Xbox Wireless adapter for windows and connect the controller using the adapter, it works perfectly fine.

Anyone having a similar issue with any bluetooth controller?