You lose too many useful skills and stats to be of any real use to your party without your soul crystal. Want examples? Let's go...

Gladiator: Good luck tanking without Shield Oath against high DPS classes. Also, enjoy your lower HP margin and no Hallowed Ground.
Conjurer: No Regen, no Benediction, no Divine Seal. Three of your most powerful skills lost in a heartbeat. Also, lower Mind.
Arcanist: Carbuncle's only, classed as a DPS so no healing potential beyond Physick. No Fester or other truly damaging skills.

The other catch is that myth armor, relic and zodiac wepaons are locked to jobs, not classes. So unless you're sporting a full set of weathered gear, your stats are going to suffer in addition to the lack of a soul crystal.