I really like the idea that a physical level represents the overall development of your character. I also love how it encourages horizontal growth. So here's my suggestion:

-make it harder to increase your physical level
-have each level up give you a very small bonus, like more hp.
-raise the physical level limit

This way, physical levels basically become a Merit Point system.