I'm not saying to remove job crystals, but this is another non-issue. Just because it's part of the lore doesn't mean the system has to actively represent that by requiring you to literally equip one every time you switch to a job. It could just as easily be implied to be used when you change jobs.