I actually started to notice a difference right after I got my second stack at level 20.

Just to clarify and to see if I got the job so far:
I start with Thunder for big mobs, burn stuff into the ground with Fire and switch to Ice to get MP by using that insta spell that switches attunment. After I filled my MP, I exchange my ice stacks for 1 fire stack and repeat?