When does this game become difficult

[Avery28]

There is a dungeon difficulty spike when you reach level 32 - Brayflox's Longstop.

Pretty much this. You will notice this spike especially as a tank or healer. From that dungeon on stuff starts to hit a lot harder, and you'll get to fairly important mechanics (Doom for example).
The next big spike you'll see will probably be post-50 with Extreme Primals. Most people are overgeared for Titan HM which was a pain at release.

I'd say stick with it. I'm also leveling Lancer/Dragoon right now and it got more fun as I leveled up.