I'm not sure how fast you're gaining levels. My first day I played I got to level 8 (I remember as I took a screenshot at the end of my first day, haha). That was by doing main story quests, class quests, and doing all the quests I had access to around Gridania at the time. I only played for maybe three, four hours as well.

But like everyone has said, it'll pick up once you get to do dungeons. Following the main story also is a good way to level. If you're diligent and play a lot every day it's not weird to get to 50 within a week.