I noticed that it only happens when moving the camera with the gamepad, with the mouse there's no jumping.