Frame rate getting stuck

I need some help.

I'm running two GTX 680's in SLI and for most parts of the game they work great but in certain areas of small towns, sections of cities, and especially in boss fights, my frame rate plummets and gets stuck at either 55 or 37. These numbers don't fluctuate. It just gets stuck there. I've tried all amount of graphics settings to fix this but nothing works. I'd understand if it was around high load areas with tons of players but like I said it happens in places like towns where no one is around. I've tried ruling everything out and have come up with nothing. Drivers are up to date, temperatures are fine. I've never had something like this happen in other games.

The weird thing is that I can go into borderless window, effectively disabling SLI, and the issue goes away so I have no idea how to fix this.