I'm starting to think it might be DX 11 and Nvidia Gameworks that is the culprit based on this thread https://forum.square-enix.com/ffxiv/threads/463738
I found out there is a way to force vulkan swapping out the dxgi.dll and adding a few other vulkan dll's (the ones people are using to make FFXIV work on Linux actually) and suddenly after a werid graphical lighting error that just stopped happening and everything was peachy keen I started instantly getting 165FPS almost solid no matter what I was doing save a few things where it was still high 80's low 100's