well, since the job gems give you a lot of stats+HP, it becomes impossible to choose to un-equip it..
for DDs the dmg is just way bigger, for tanks they get a lot more HP, and healers heal for more and got more mana.
so.. its not just swapping 5 job skills, you would also lose a big amount of stats!

If it wasnt for the stats, i would only equip my gem during dungeons.