Your item level is the only thing that matters for battle retainers, but gatherers solely look at your stats, with item level meaning nothing. Gathering is what determines the rare item threshold as well as the amount of items brought back from regular ventures. Perception increases the amount of HQ they bring back as well as reducing the amount of time it takes for them to do a venture. You saw no real effect from equipping battle accessories on your retainers because gathering accessories tend to not have any stats on them but GP.
Also, your level and class mean nothing for quick ventures, even if sometimes it feels like it does because your monk retainer brought back a pair of claws once or twice. Any level and any class retainer can bring back any "non-rare" item. And once you meet the requirements, the rare item pool is the same for any class as well.