A permanent magic vulnerability wouldn't be much different from the permanent physical type dmg that the physical classes have atm and that is the idea behind it

Also personally I don't think it would make casters that OP compared to other comps