The issue is that it penalizes you heavily for having your avatar in the action, where it needs to be in order to give favor. The two minute charge up for a full powered favor is an outright eternity on anything but the longest of fights, and anything that lasts that long generally has enough AOE potential to kill an avatar long before it ever caps favor. Having Avatar's favor in and of itself is more than enough to keep an avatar out (if it started at full strength and did not have chargeup)
You are aware that Geo has elemental damage spells, right? They're made to compliment their buffs with straight up nuking. They also don't penalize the Geo at all for buff potency or duration outside of MP management.