All you need to do is have strict drawing protocols when entering a fate circle. (like above mentioned)

On top of this you should remove all unnecessary mobs/NPCs whilst a Fate is in progress, de-spawn chocobos companions and minions, the first thing not to be drawn should be summoners pets, as long as they still function as normal (whilst invisible) that is just fine.