Maybe different abilities are affected by affinity differently. My Cures and elemental spells don't feel significantly weaker with One With Nature equipped, although the CD is about a second longer. That's very different from double cd on Punishing Barbs. Maybe this is something we need explicitly explained like the gathering fatigue.