The character auto-attacks are always the base physical attack, so the casters will do almost no damage while the non-casters will be hitting like normal.
The caster weapons list the magic attack value in the tooltip but there is a hidden physical attack value that is much lower.

Lesson: if someone is going to get confused, have it be a caster.