(PC) Geting very odd frame rate fluctuations

My Systems is:
Intel Core i7 3820
16GB DDR3
2x 660Ti (SLi)

(I generally play with adaptive Vsync, but for the sake of the post im using my non-Vsync fps)

I dont understand the odd framerates I'm getting in this game. When im out and about in the world I might get anywhere from 30-150FPS, but generally speaking ill sit around 70, which is just fine. However when I go into any major city except Limsa, my FPS drops into the high 30s to mid 40s. However when I'm in Limsa in particular I'll drop into the 20's.

I don't understand this at all, neither of my GPU's are being fully utilized, and neither is my CPU. I have a friend with a core i5 that is less than half as powerful as my CPU and he gets close to 60fps in Limsa, and he also told me he CPU usage goes up to about 85%, whereas mine will not go above 50% no matter what I do. I also have a friend with an old Phenom II X4 955, which is miles behind my CPU and he gets nearly the same FPS I do...something is wrong here.