While you can certainly start at anytime, it's definitely easier once you have a retainer to store items on. Crafting, mining and harvesting will eat up inventory space fast. You get a retainer shortly after you finish Copperbell Mines in the story.

I do recommend working on multiple crafts simultaneously. You'll find items you need for one can be gained thru another. Makes things easier. I have every craft unlocked myself.