What you've built up are physical levels and weapon skill ranks. These will be used in conjunction with the job system, so no worries about your hard-earned EXP/SP going to waste.