I decided I won't be buying any more retainers than the 4 I already have. Inventory is a bit stressed, but that's because I'm a glamour addict. But it's not un-manageable. One rule I imposed on myself is only storing HQ versions of common lowbie crafting items (and even higher level ones) - because I'm either carrying the NQ version on hand for the most common things, or I can buy it from an NPC in about a minute if I really needed it. The exception is my giant pile of gems, ingots, and ores because I'm a primary goldsmither and one retainer is dedicated to holding all my rocks. (She also gets all my materia.)

4 retainers lets me have one of each gathering type and gives me adequate storage. I really don't need any more than that.