As stated by the notes, their average item level affects overall results on what you obtain (especially in terms of HQ), exp I'm not sure yet. So sending your retainer out in event gear and swimsuits will do far less for you lol (even the quest retainer stated this basically.)