Note that that applies only to gathering retainers. Since the score they use is their perception score, and all gathering accessories give only GP (unless melds) they get zero benefit from accessories.
Combat retainers, however, are about ilvl. Leaving them with completely empty spots on their right side really hurts the iLvl. Note that you can give them any accessories you might have at hand, it doesn't matter if you kit a rogue with healer accessories, as long as it increases their iLvl enough for the requirements they need.
