Not a fix, however if you:
a)turn off the controller in the background option in xiv config.
b)run xiv in windowed mode, not full screen.
c)make sure xiv is not the active application when you go afk.
d)turn the controller back on before making xiv the active application again.
You will not have that issue. It's a pain of a workaround, but the only one I have found sofar.
FYI, I do think it is more of a controller issue. I have the same issue in Vent (I have my push to talk mapped to my controller as well).