Random thought on changing buffs....

Buffs don't expire on time. Once you activate them, they're permanent until either you deactivate them, or you get a status that deactivates them. Instead of time, they take up a chunk of your stamina gauge, effectively lowering the maximum. Different buffs lock out a certain amount of the gauge, with lower level buffs locking out less than higher level buffs. This forces you to balance what buffs (and levels of them) you have up with how frequently you want to use attacks.

You could also make it so that a buff cast by its native class takes less of a chunk out of the recipient's stamina bar than one cast by someone who just cross-classed it.