Game probably is not optimized for nVidia cards, so I heard in other threads.
I have the same issue, randomly losing a bunch of FPS in random areas regardless of players number/activity. Just me and a whole bunch of nothing and I still lose about 20~25 FPS, even in cutscenes there's a small lag.
Some of it is not even counted by the in-game FPS counter.