Personally when i see this, I don't think of my retainer going out into the open world to gather stuff from gathering points.
I think it would apply only to the garden you have outside of your house (stuff you grow yourself in your backyard).
Anything else imo would be kinda silly.