Gathering Retainer Stats: What matters?

Is it PURELY just the Gathering that's important for the retainers or does Perception count also? Oh and GP (for BOT and MIN at least)?

Excellent question. I too was wondering about this.

Gathering is the only thing that matters. You only need to hit 408 (IIRC) to be able to send on the highest level 18 hour ventures. Perception and GP mean nothing for your retainers.
Each venture has a loot table, which your retainer randomly gets an item from. Thus, more Gathering does not affect the % chance of getting that rare item.
FYI, it is common practice to put none gather jewellery on your retainer as they have higher iLvls and can have higher grade gathering materia melded to them before hitting item stat caps; Fishers also do this but also meld Perception as GP is useless to a Fisher.