Not only is it a case of lower MND and lack of traits, but also damage rating of the weapon. Each weapon has a Physical Damage and Magic Damage. You only see the one relevant to the class on the weapon's tool tip, but the other one IS present as a hidden stat, and can be found via data mining. Typically, a physical weapon has a slightly lower magical damage (Spiked Cobalt Labrys has 34 physical and 25 magic damage), while the opposite is true for magical weapons (Jade Crook has 45 magic and 29 physical damage). So, there is a noticeable difference trying to use spells on DoW or physical skills on DoM.

You'll find Protect and Stoneskin (especially applied outside of battles) is more useful as a cross-class skill since it does not depend on MND. You can also use Stoneskin as a "cure" of sorts, since there's functionally little difference between mitigating 10% of your max HP in damage, and healing for 10% of your max HP. CNJ obviously gets better versions due to traits, but it's still something to consider.