Audio stuttering and FPS drop while on FFXIV

I have heard a few of my friends alongside myself having an issue where under certain conditions (not sure what yet) the game will slow to 1-5 fps and the audio will stutter alongside it. Usually this happens within the first 1-5 minutes of playing but sometimes it lasts longer than that for a few hours.

I'm running a Nvidia GTX 980 TI card, and my friends having the issue reported nvidia cards as well (not sure what one)

I reinstalled windows entirely, reinstalled ffxiv, reinstalled drivers, tried everything. It only happens when I'm playing FFXIV, but I mostly only play FFXIV so I don't have a lot of data with other games.

It occurs in both directX 9 and directX 10

Does anyone have experience with this and perhaps fixing it? I'm not sure if it's my graphics card or other components that are bad, I ran some diagnostic stress tests and no errors showed up, but FFXIV is currently almost unplayable and super unreliable for me and I don't know what more to do.