I use a 360 controller and have the same problem with the right analog stick (mapped to the camera). Every now and then the camera will start slowly turning, and I then can't move the camera at all with the keyboard controls. Of course, tapping the analog stick fixes it, but it is kind of annoying.