Occlusion culling broken on PC?

I have a very strange and annoying issue on the PC version of the game. I run at a smooth 60fps (with vsync enabled), but sometimes when I turn the camera, my framerate will drop dramatically, and then slowly recover. This is very prevalent when in areas like the Gladiators Guild in Ul dah, when turning in the direction of the fighters. If I don't move the camera, my framerate is fine- I can walk up to 25 ish players in Ul-dah and keep a decent framerate! Some other players have reported this problem in a now-closed thread, but it doesn't seem to get much attention. The reason it happens may be because the game is not removing the models, textures etc from memory, or the feature is simply unoptimized. My specs are:
CPU: AMD FX-8350 Octa Core Processor
GPU: ASUS Nvidia Geforce GTX 670, 2gb VRAM, DCU2
RAM: 8GB Single Channel 1600mhz
Storage: Seagate Barracuda 500GB HDD
Motherboard: ASUS Sabertooth 990FX R2.0
PSU: Corsair CX500W

This seems to mainly be a problem with Nvidia cards, if anyone else has this problem please comment on this with your specs, I might put something like this on the Nvidia forums in case this is a driver issue.
Here are some links which may be useful:
http://forum.square-enix.com/ffxiv/t...-Comprehensive
http://forum.square-enix.com/ffxiv/t...-angle-any-arc
http://forum.square-enix.com/ffxiv/t...m-also-for-AMD
https://forums.geforce.com/default/t...ive-/?offset=2