This doesn't happen to me. I stopped running Castrum a week after hitting 50 as a Pugilist. I suggest you stop running it, too.