It depends on what job you're playing, but several of the class/job quest battles can be challenging, particularly if you're a rather fragile caster.
I also found the next two primals you will face (Garuda, Titan) to be hard when I did them shortly after launch.
Otherwise I'd say.... high level crafting.
Which is to say... different players have different ideas of what "difficult" is:
There are a number of battles, especially after lv50, but also in some earlier dungeons and trials, where you need to pay attention to mechanics - this is probably what I would consider "difficult" and also "fun!"
There are certain beast tribe quests (item transport quests that end if you are hit) which I managed to fail multiple times, but they seem to be easier now. (lv40+)
There are also activities that somewhat challenging puzzles (at least if you solve them yourself) like triple triad, riddle quests, and high level crafting. (varying levels)
There are certain FATEs (Odin, Behemoth, Gorgimera) which can usually only be won by large forces and/or can take a while to complete.
There are other "grindier" forms of difficulty such as:
Quests that take a long time (in Earth time) to complete, such as the beast tribe quests (which I actually enjoyed.)
Quests that require you to travel to various distant parts of the world (fortunately you can teleport there usually) only to find that you need to go back to where you started. (Every story quest, and lots of others.)
Quests which depend on low item drop rates or a high failure rate, such as getting particular dungeon drops, advanced materia melding, or card collecting.
Quests which require a large number of repeated steps to complete, such as the relic weapon quests (which may also involve collecting lots of items), collecting tomestones or other tokens, or leveling crafting or gathering classes.
Quests where you may have to wait until the right Eorzean weather or time (e.g. catching certain fish, or gathering certain items) and/or for passage of Earth time (e.g. a-rank hunt timing and s-rank spawn conditions.)
Quests which depend on certain FATEs, which may take a while to spawn (and which you may miss if people finish it too quickly.)
Quests which seem to depend upon the actions of many people on the server (e.g. Doll festival quest) - this is a neat mechanic which I would like to see more of!
Quests which have too many people competing for them (e.g. hunts in patch 2.3 and 2.5) at once. (Fortunately hunts seem to have recovered as of the current patch 2.51.)
Chocobo leveling.