If you're looking at it in terms of how much skill "matters" in terms of DPS, then SMN/BLM are probably the top two. SAM can be hard to play optimally but the difference between a good and bad SAM isn't that large.
OTOH, a bad SMN might do less than half the DPS of a good summoner, and the same can be said of BLM. It's very easy to notice good and bad BLM/SMN while pretty much every SAM will do high DPS regardless of skill - some will just do a bit more damage because they read the skill descriptions.
For SMN especially, a single death tanks your DPS for the entire fight, they have probably the most oGCD-usage, and their rotations change every single fight depending on procs/phases/movement required. Same for BLM but they recover from death better.
