HQ makes a minor difference if you're considering the Gathering stat only, since that is the only stat on gear that a gatherer uses. Also, 'full set' should only be those pieces of gear with the gathering stat. Perception and GP are useless to your retainer.

For the hunters (combat jobs), a full set of HQ might make some difference when retrieving rarer items.

I've somehow managed to fetch almost all the rarer items with the gathering stat just over the minimums required. And yes, fishing retainer is the one who gets the fat cat minion.