I kept the vast majority of my AF stored in my Armoire, however, I then proceeded to do some of the AF-related quests for my Jobs. This resulted in me having two sets of AF, one stored in my Armoire, and one stored in my inventory/armory chest. I can't store the duplicate set in the Armoire, and I can't discard the duplicates either so I'm stuck with the second set eating up inventory/retainer space.