I don't want to see Physical level go away. I do however want the stat points to affect me more than it currently does.

I think something else could be done with this, and probably slow down the physical experience gained as well. Doing away with it just feels like a little less more customization with my character.