I think it's your hardware. And I would like to share my experience.

I'm playing on a laptop with 2gb gt555m (144 shader cores only), i7 processor and 6gb ram. The graphics and gameplay is smooth (laptop standard settings), but when I'm in marketplaces or towns with many players the fps drops dramatically.
That's why I'm selling this laptop to upgrade to higher specs. I'm planning to buy a laptop with video card 2gb gt765m (700 shader cores) because can't own a desktop at the moment.

And yes in order to play and enjoy FFXIV you need a good machine. But if you are a casual gamer you should be fine with my specs. But with your specs I think your laptop will have a hard time. Let's upgrade!