If you want to perform close to the highest possibile level, SAM is probably the most challenging melee. It has an extremely high skill ceiling, but a very low skill floor.
BLM has a very simple rotation, but requires a lot of effort to execute this rotation in relevant content because of its limited mobility, so it is generally viewed as a very complex job to master as well.
Both are very selfish, with little to no utility for the rest of the party, and have a very high personal dps and are generally welcome in most parties.

Then we have MCH, with a moderately high skill floor and a very high skill ceiling, but right now it's in a pretty bad spot since it's outclassed by brd in every possible way and scenario while also being harder to play at a high level. But if you're looking for a challenging job, MCH is a good pick and it's very fun to play.