If it were not intended to switch back to classes, SE would only allow you to remove the soul crystal when you switch to a class for which you have not equipped the corresponding job. Being that you CAN remove the soul crystal almost any time you like, it IS an intended mechanic. Therefore, it's not an option I'm willing to ignore should the situation arise where it would be useful to switch to a class.