Your first 50 takes a LOT longer for a variety of reasons - story missions, lack of resources like grand company seals and gil, to name a couple. Once you have a 50, know the game, know where to go for what, know more people who can help you with stuff and you have resources to gear up easily along the way, things speed up significantly. Technically, you could grind fates pretty much all the way but that's neither fun nor educational (you don't learn how to play your class very well from fates). Don't worry about it and enjoy the ride.