Things I think need fixing up or changed.

[dejavutwo]

Is there a magical secret because I've tried everything.

I don't know how to do it on a controller, but if you right-click and drag you can pivot your character in-place. There should be an equivalent on the controller. Just make sure that when you are rotating, you are rotating your character, and not just your camera.