I would think we would just lose the exp earned for physical level if they go away completly and seriously I don't think anyone would/should care.. it's not like it's hard to actually level it since every SP you gain on any DOW/DOM/DOL/DOH gives you exp for your physical level. You wouldn't "lose" anything because the stat points will be gained with class ranks.

There is nothing unfair about losing them, if you leveled a class to X rank, you'll get X rank worth of stat points to allocate for that specific class.