Sounds like both tanks aren't raising enough enmity. How's your Weapon level compared to the rest of the party?

Try saving FoF for the swap and as the above poster mentioned, que up savage blade to get an attack out faster.

If you're safely taking big hits, try using STR accessories to help keep raising the aggro table through out the fight.