Keyboard bindings don't apply to the cross hotbar that's activated with the controller. They apply to the "regular," straight-line keyboard hotbars that you probably have hidden. The first two regular hotbars also happen to automatically fill in with a class's skills in the order they're learned. So what you're seeing is a hidden hotbar activating on keypress, not skills activating out of the spellbook.
You can show the hotbars in question through either the HUD layout (click once to select a given item, right-click to make it visible; hidden items are blue in HUD layout), or by going into the hotbar section of the character config menu and ticking their checkboxes. Once they're visible, you'll be able to see what's going on, as 1 through = are displayed in order, left to right, on that hotbar.