2 retainers is more than enough, just get rid of the muck you don't need. Their biggest issue is glamours atm, just get a glamour book that you apply glamours to when you obtain the item so you don't need to hold on to it. Would save on their server resources too.

Guild Wars 2 has used this system for years. Saves so much space. Personally I only use the first 4 rows on my first page of inventory and a good portion of my chocobo saddle for totems/relic/materia holding. My retainers are all practically empty and have been since ARR.