I have never had a problem getting enough shards or crystals. Many crafting quests and levequests give them as rewards which gives you a nice base to start from.
Once your retainers can start gathering them you can just send them out for a batch or two whenever supplies start to run low. Send out 2 retainers and you can get 120 shards in 40 minutes - which is more than I use up by crafting during that time.
You can also go out and gather them yourself if needed - it is trivial to do so, just takes time. This should rarely be needed however.

Clusters are harder to get in large quantities, but there aren't many recipes which require them so they are not much of a problem either.