I've assumed that the damage you lose out on them not being used under the effects of RoF/IR is gained back by using them more often, but don't quote me on that.
Anywho, I think Brotherhood is best used after/as perfect balance wears off. By that point you've got things like Trick Attack going off, and people have gotten off some of their buffs and the like (Slashing Res Down, Piercing Res Doen, etc), so they'll make the most of the time.