'Most challenging job' is probably too subjective to get a proper answer for.
In terms of desired jobs, in terms of DPS a Bard is pretty much guaranteed a spot in any raid, but it's also easier to do well at than most other dps jobs. I think Samurai and Ninja are both well desired too, and as the previous poster mentioned these are more 'difficult' dps jobs