The colors do come from the crystals. so Cyan, Blue, Green, Orange, Red and Purple have associated elements.

The poles also seem to be associated with Black and White if the poster Urianger has of the elements and the first anima step are any indication (though I headcannon an association with Magenta and Yellow to keep things chromatic and because the canon association assumes something that we now know is the opposite case [That being Astral correlating to Light and Umbral to Darkness])