Drift is fine, especially when the optimization of the skill, in itself, that would otherwise drift affects what is net-optimal for one's rotations, allowing for further rotational variance and skill expression than would be present with a fool-proofed / nearly non-optimizable skill.

If not for the MP costs of opposite element casts being reduced to 0 under AF3/UI3, I could see a place for a reduced CD on Manafont, but I do not want to see it handled through what is effectively an additional partial charge per your suggestion, nor foolproofed in any other way.