Prior to 2.1, Astral Fire and Umbral Ice were only removed and applied when a spell hit. However, in order to fix the double dipping into UI3 that Yoshida stated was allowing Flare to be cast twice, they made it so that the buffs are removed on cast (in the case of going from Umbral/Astral to Astral/Umbral) and applied on hit.

Sucks, but that's how it is.