Both methods end up being about habit. Like you said, it's all about muscle memory.

When switching from controller to keyboard, you'll be slow.
When switching from keyboard to controller, you'll be slow.
For keyboard and mouse, an MMO mouse can often help with movement as it frees up the left hand to do movement. When switching to this, at first you will also be slow.

Pick something and stick to it until you're not slow anymore.

As far as playing efficiently on controller, I recommend the following sites.
https://supernova-squirrel.herokuapp.com/
https://www.reddit.com/r/ffxiv/comme..._general_tips/

And learning how to effectively use filters. These are in character configuration.
And going over your controller config (system configuration), especially for the Analog click-in. Using the macros #98 and #99 is often better than what the default options are (especially the one to reset camera...)