I know what you mean, of course they would have to cap it of you switch to a low rank class.
Actually I like it that once your physical level is high, you are stronger when you start a new class
However the physical level should be a lot harder and slower to level IMO.
But as i said, i would be fine with other solutions.