To be honest, the circumstances under which we acquire crystals don't seem to have any sort of commonality. Sometimes we're given the crystal, sometimes we find it, and in Garuda's case, we rip it out of her chest. It's like the universe is saying, "Welp, it's been a while since you last got a MacGuffin. Looks like you're due!"

While several crystals are found after defeating Primals, it doesn't really seem like those crystals are tied to the primals in any special way, in spite of the fact that the crystals we got from them were the Primals' element - particularly given that the Ice crystal doesn't even HAVE a Primal. (Shiva's first summoning was well-after we collected the Ice crystal.)

Not the best example of storytelling in this game, imo.