A lot of the complexity in how this effect would work can be solved just by ignoring how this worked in FF console games and simplifying things. This whole "reflect a reflect" nonsense, inability to cast cures on members, etc. is too much complexity.

Make reflect work to only reflect spells cast by foes back to the foe that cast it. In order to not have it overpowered, reflect should not be 100% but have a % chance determined by the enhancing skill of the caster compared to some stat of the foe. For weak mobs, the effect % could be quite high. For though NM's, the % would be lower. For AOE spells, only the target of the spell would reflect back to the mob although others in AOE range would have the same chance to mitigate the effect of the spell (misses you but you don't reflect the spell - only the target does).

Having a % proc rate would also open the door to specific gear enhancements that can improve this or job traits that add to this effect (for example, give the spell to many mage jobs but give some jobs trait that improve the reflect rate) or merit categories that improve the effect.