Horrible performance

o7 Everyone,

So I've recently built a new tower that I thought would be able to handle this game but I'm getting awful performance. We're talking 9-15 FPS when anything substantial is on the screen. The character creation and loading screen is smooth as butter but as soon as I get into the world and move around for more than a few seconds my fps tanks from 59/60- down to the teens. It doesn't seem to matter if I have maxed out settings or the lowest settings possible either. I've been searching for a long while and can't seem to find a fix. I'm starting to think it's my graphics card because it only has 2 gigs of VRAM.

What I'm currently running;
Motherboard- z97s sli krait edition
cpu- i5 4690k at 3.50 Ghz
gpu- GTX 960

(this is where I think the issue is, I might need to bump it to a 970 for more vram, I also notice playing other games run butter smooth until I go to heavily cluttered areas with lots of crap like heavy grass etc, my performance also tanks if I try to watch youtube while playing a game, also telling me it might be a VRAM issue, the thing is it's 2 gigs VRAM but from what I hear it borrows 2 gigs from your normal ram too, and I also hear this game even on max setting only takes up about 1-1.5 gigs of VRAM so it's not like I'm maxing out... or am I?)

annd 8 gigs of ram
windows 8.1
GTX960 driver version 347.52

If I could get any help it would be greatly appreciated. Just bought the game and it looks absolutely gorgeous and I'm a huge final fantasy fan so I'd love to actually get it to playable standards...