I don't understand why this is restricted in the first place.

"OMG! This evil person came along and made my crops grow faster!"

Yeah, right.

Make an achievement for it. Call it "Mass acts of fertility" I/II/III/IV and give titles like "Fertilizing fairy" up to "Ultimate Karmic Gardener" to people that fertilize/tend like 100000 plants not in their own gardens. Maybe give out a minion that is a bag of fertilizer with arms, Etc.

Also perhaps make a new fertilizer that can only be used on others gardens that speeds up growing by 2 hours instead of 1 with a shorter cooldown.

In my opinion there need to be more opportunities for altruistic behavior in the game and this seems to be a huge opportunity for it.