The cast time and MP cost of Fire 3 is too big. You only use it to get the 3 stack Astral Fire buff and when you have a Firestarter procc, wich removes the MP cost and cast time from Fire 3. Same for Blizzard 3, you only use it to get the 3 stack Umral Ice buff.
Fire 4 does not extend the timer of the Astral Fire buff. So you have to use Fire 1 after 3 (or at most 4) Fire 4 casts or your Astral Fire buff will run out.