Leaving Castrum (or Prae, for that matter) is fine.
Just turn off 'in progress' queuing for the MSQ Roulette. Or everything's going to be a half-empty Castrum 0 minutes in progress, and move on with your life.

People get a penalty if they're the first to leave. Which is enough.
And if they're DPS, they're still staring down the barrel of another 15-30 minute queue on top of it. Which is probably going to be Castrum. Again.

The only thing that *isn't* fine is just standing around idle, either at the entrance or on /follow, hoping 7 people will get your XP for you.
I've tried to initiate kicks for THAT.
I even reported the one guy who was dumb enough to brag, in chat, about what he was doing. Since that would be an easy punishment for the team.