Okay, did some more testing. Game ran fine with Depth of Field on and Ambient Occlusion Off. The moment I turn Ambient Occlusion On though, it doesn't take long for the game to crash.
With Ambient Occlusion Off my GPU laughs at how easy it is to render the game. With it on my game hovers around 45 but goes as low as 35 and as high as 60 depending on the area. Doesn't matter if I am on my 680 or 580.
Something I still need to try is put the game at 30FPS and see if it crashes with it on. Something I also noticed is I can be out in the middle of no where (was at Iron Lake doing some farming) and with AO on I never crashed once in a span of 8 hours moving doing Botany and being AFK at the Camp. Not more than 30 minutes in Ul'dah did the Game crash due to losing the DirectX Device. This was with AO On. Turn it off, no crashes.
So we definitely need a driver fix.