Well, IMO it's correct all recasts reseting after class change. It would be kinda unfair having everything ready right after changing classes.

But what Sigmakan said is also valid since class-exclusive actions do not use action points if equipped in a different class.