It depends on the class, but you're mostly going to be missing out on important skills. I've had marauders and gladiators show up in level 35+ dungeons, and without their defensive and enmity abilities, it's practically hopeless. Of course, those players typically don't know about jobs in the first place, so the job crystal isn't the only thing causing trouble.

Archers can get away with it if there isn't much need for AOE or singing, conjurers can handle easy content since they only lack regen and shroud of saints, and I might be mistaken but I think most of a thaumaturge rotation comes from level rather than job quest.

The downside is that you will simply be missing key abilities for every class that improve your role effectiveness 20% or more. That's probably why people look at you poorly. You sacrifice targeted effectiveness for RP value, and in random groups that simply want quick completion, that doesn't make a lot of smiles.