It's not possible to discard the crystal in any way so once you obtain it from the job quest, you keep it forever. You didn't deposit it to any of your retainers did you?