Why not just make it so theres a cooldown on lost actions when you swap them out, and switching out lost actions removes all buffs/debuffs received from them?