If we can swap without cooldown penalty in Diadem, why not in the overworld as well?

[Paladinleeds]

Then eat the CD (the class change one) if you need to change, just like everywhere else.

And in the time that cooldown happens, everyone else has got a significant lead on that star rank meaning your group get no chest from it and people get pissy and leave... Or here's a brutal idea, if you change classes in Diadem, you get hit with a 200 year cooldown on all abilities. That'd stop class changes in Diadem wouldn't it? And the cooldown would affect every DoW/DoM/DoH/DoL ability (and your ability to gather on BTN/MIN) would be affected no matter what class you changed to.