The thing is, if you actually play the mage part of your job; Then you are casting spells at least half the time, which means that you are not swinging your sword, which in return is a considerable loss in DPS.
Then there's the part where a proper mage casts spells in optimized gear to make sure that the spell does not get resisted on your enemy and in case of buffs, that you have an casting rate which is as fast as possible and at the same time has an as fast recast as possible because in the case of buffs in a proper setup you should be casting the spell on the next person.
Proper casting gear is not melee gear, which means that for the split second you are swapping back to a melee set you'll likely miss half the strikes.
Unless you are one of those that believes that swapping gear is cheating of course.
I'll simplify above a little for you.
Red
mage -> Spellcaster
Let's look up a little description of the job Red mage:
http://wiki.ffxiclopedia.org/wiki/Red_Mage
Time to use your sword?