Number of players online should have nothing to do with fps issues in dungeons... In towns if your PC has to render more players, sure. Guessing you've already tried updating and rolling back drivers for your GTX670? What are your other PC specs like? If you're getting massive stuttering like this even in dungeons, I'd definitely try updating GPU drivers or rolling back if there are no updates.