I think I found the issue (with the 5700 series at least). It has to do with the 90 fps cap the game had when Shadowbringers launched but ended up removing. Basically FFXIV becomes unstable at high fps with this card and you need to cap fps at 90 or lower. 90 is what the game's internal testing found to be stable..

This lines up with what I'm seeing in game. I had Chill set with min 50, max 100 when I first posted. This kept my fps under 90 almost all the time. The crashes mainly happen in PVP, healing alliance raids and when dealing with retainers. With how Chill works, this makes sense. These are scenarios where I'm moving my mouse a lot so my fps would spike up over 90 and cause instability. This would explain the randomness of the crashes, why downclocking the GPU seems to help and why FFXIV is the only game giving me issues. Once I capped my fps at 90 instead of a 100, I haven't crashed since (4 days and counting!). I'm still on driver version 19.12.1.