I think I have found a solution.

It seems like Steam is the culprit even while not playing the steam version of FFXIV.
I connected a Hori pad for Nintendo Switch which is not recognized by Steam and it worked in FFXIV. So I quit Steam and terminated all its processes in Task Manager and FFXIV started to react to all my controller inputs again. PS4, Xbox, Logitech and Switch controllers all work like a charm now.

Anyways, hope this might help someone else.
Cheers,
Ake