Exact same problem here.
P2~P3 was smooth as butter, haven't changed my rig and in P4 got a huge FPS drop and regardless of graphic configuration in-game. Maximum or minimum, the same FPS loss in random places, even in cutscenes with 2~3 NPCs just talking to each other.
In the old beta forum there was a rather large thread about this very same issue and I already saw 2~3 topics here about this.

GTX 680 (no OC)
i5 2500
8GB RAM