It's your processor. I have the same graphics card (GTX 970) and a 1st generation i7-920. Generally I get a decent frame rate that I'm happy with (~60), except for Idyllshire. It's all the people and different combinations of outfits. The processor is literally having to "assemble" all those different people/outfits, they're not single objects, but the game engine has to build them. Race, size, wearing this top, that hair, this colour...

I don't know about your server, but on mine there's a lot more than 30-40 people there (you said that in reference to hunts) and it's more like a few hundred within visible range at any one time.

So yeh you could upgrade your processor and it might "fix" it, but probably something easier to just live with.