Difficulty, I think, is based on each individual. One person might think that this class is harder than this class or vice versa. Every answer here is an opinion and people will disagree and argue over and over.
For example, a majority of people rank Summoners to be below either a Monk or a Dragoon but to me, I find Summoners to be the hardest class to play simply because I don't want micro-managing any critters to help me do my thing.

Bottom lines is, It's all subjective and it's up to you to decide which you think is the most difficult class to play.

PS Only one thing here is a facy, Bards are the easiest to play (: