A controller makes you slower. Even though you say you have 'no problem', using a controller is quite a bit slower than hitting keys.

There is one thing that could mitigate this. Efficient and fast macros.

But the macro system in FFXIV is .... indescribably bad. Sometimes I find it easier to just equip my gear myself instead of run my macro.

In FFXI I was always a bit slower using a controller but its macro system was so fast and nice, I didn't mind so much.