The original intention was the level them all. Crafter Souls only came around in Heavensward.
Other than a bit of bonus stats, I don’t even use the specialist stuff myself.