To the contrary. The term "crystal" actually has great geological significance.
When a body of magma cools to form an igneous rock, it undergoes
crystallization. Yes, that is the technical term. Rocks are made up of lots of minerals, their crystals interlocked. The slower a rock cools from magma, the larger its crystals will typically be, helping us learn more about how, when and where the rock formed. There are some crystals that got freaking huge throughout history -
this article shows some of the largest crystals found.
Crystals are made of rock. And stones are essentially pieces of rock.
(There is a more technical definition, but I won't go there.)
I wouldn't hesitate drawing a connection between "stone" and crystal when discussing the "lodestone".