The long cast times and the AF/UI timers were interwoven mechanics to a degree; remove one and you lose an aspect of the other.

No timer gives no pressure to risk just one more Fire IV cast before refreshing with Paradox or Firestarter. Low (or no) cast times reduces that pressure to get that cast out just in time (and in turn also reduces the value that Spell Speed used to provide).

If we start removing mechanics because failing them ends up too punishing, you can very much extend that to many other mechanics. Failing to DPS properly punishes you with enrage, so why not get rid of the enrage timer? Failing to mitigate can kill a tank during a buster, so why not just remove mitigation and reduce damage from bosses to compensate? Not healing can result in the party's dea-- okay you can see where that's already starting to no longer apply with how easy it is to drop healers in a semi-competent party. These are extremes of course, but that's the end result of "removing punishing mechanics". At some point, you have to have pressure to play the content or job at hand.

It's one thing to reduce some punishing aspects in a jobs kit to keep the skill floor low, it's another to remove the punishment entirely.