Is Samurai a difficult class to play?

In principal... no, not really. The rotation isn't difficult, but I find it tedious because I don't enjoy it. Though the difference between a good SAM and a mediocre SAM is huge.

The basics are really easy : 3 combos, 2 positionals and your kenki gauge to manage. It's probably one of the easiest class to get into, with RDM.

When it come to min/max thought, it's probably as hard as DRG/MNK when you try to optimise your damage on a boss. NIN is still more difficult to handle for HL plays.

If people think that SAMs are retarded, it's because 90% of the samurais you will see in game have absolutely no clue how the job works (to the point where you wonder if they actually read the damn skills) and got a really bad DPS, thus gaining a bad reputation with the community (and the meta-only idiots).

Every class in this game is fairly simple to play at a basic level and complexity increases as you explore the intricacies of the role. As long as you do a bit of research on the job and practice on a dummy/while leveling I see no reason why you can't pick up any job and play it if you wish to. I think the latter point about while leveling is something that alot of people don't do but it really does help. Personally, when I pick up a new class I look into it first and set my hotkeys up so everything is in a familiar and comfortable spot without having to rearrange it for new skills as I go along. This lets you start to develop muscle memory and familiarity with the way you're set up in a much less pressured environment (leveling dungeons).