"fall off hard" is a bit of an exaggeration.

At 90 percentile on the final omega fight, for example, a black mage does 8500 ish, and a red mage does 7500 ish. A difference of 12%, comparing a caster that is 100% entirely selfish, with a caster that has higher mobility and provides a party-wide damage buff. and the rez, but that's not so important when comparing parties that can compete at 90+ percentile.

Yes, they're the weakest caster, but no, they're not so weak that they can't compete. They are, in fact, 6th place among all 9 dps, staying ahead of bard, dragoon, and ninja. And the differences between the jobs aren't even that severe.

Bottom line, all classes can compete reasonably at any content that you could hope to do. Just play what you want, and if you play it well, you'll clear.