Some more info:
CPU is at 70-80% utilization in Limsa. FPS drops to 32.
CPU is also at 70-80% in Gridana, FPS drops to 40 but then goes back to 60.
Same CPU utilization in Ul'dah, FPS drops to 50.

An FC mate who also has a 970 tried to OC his CPU from 3.8Ghz to 4.3 Ghz. There was no improvement on the FPS. We think it's an optimization issue rather than a GPU or CPU problem.