When some classes need to gain more than others, it shows two things. One, that the gain itself will not be fair (which is why your proposal is inherently unfair). Two, that the existing situation itself is unfair.
The only way to kill both of those unfair conditions is to simply gain all of the other cross class skills as part of becoming a Specialist.
To keep them from becoming *too* powerful, get rid of the current Specialist-only actions when you get rid of the Specialist-only recipe flag. Any weaver, regardless of Specialist (or lack thereof) should be able to produce any weaver-made item that there exists a recipe for.