SAM is a point blank black mage. It's how the class moves and feels and honestly feels AMAZING. It sounds like to me that it's just not a class you have synergy with if you're not enjoying the casting of Iaijutsu.

Yaten and Gyoten are honestly not that necessary early on. It's all early content and SAM barely touches it considering it starts at 50. Removing kenki cost is kinda pointless and removes the tactical considerations of the abilities.

Hagakure was improved into Ikishoten. It was a rather pointless ability in my opinion, messed up my rotation if I tried to use it.

Meditation is amazing, Shoha feels weak but it just makes an amazing ability more amazing. A sign of a good SAM is how fast they enter meditation between phases of a boss fight.

Kenki honestly floods in if you understand SAM, I have trouble managing it at times due to overflow. If you get the positionals, kenki is absolutely raining down.

SAM is a class that has the intrigue of operating like a black mage at times and being quick at other times. It has a means to close in quickly (enpi+gyoten) and is intended for 50+ content. The casting of Iaijustu is the trait of SAM that make it a unique class and adds to the feel and sense of power. Removing the cast time would mean that the damage would need to be reduced as well. A balanced skill judges time, output, and risk together, removing a risk or requirement would require it to be also be balanced into a lower output. That said, MCH and BRD both became like black mages during Heavensward and it was TERRIBLE. Every attack was a cast. This was changed for 2 reasons. First of all, it did not flow with a class not intended for it and was overused. Secondly it made them operate as casters, defeating the purpose of separate classes. SAM is different because it uses cast attacks on occasion and does it up close. Being able to manage this makes it one of the top scoring DPS in the game.

If you don't like the class's quirks, you are probably not going to be able to properly enjoy that style of class. Trying to take away the Samurai's cast is like trying to take away the Black Mage's cast time. Players of both classes generally prefer gigantic damage with cast times rather than weak attacks with none.