You didn't spend any extra time and energy getting your physical level up. It leveled up concurrently with your class level. There is no way to only level one or the other.