5.
It started back when they first introduced the additional retainers service, because I was hoarding all kinds of crafting materials, both HQ and NQ. I had two combat retainers and three gathering retainers.
Then I think it was in 4.x when I decided to get rid of all the crafting materials, and I started hoarding leveling gear. One retainer for each main stat (VIT/MND/STR/DEX/INT).
Now in 6.0 I'm thinking of dumping those leveling gear as most of my jobs are over 70, with only two or three under 70, meaning I could use Trusts for leveling and get all the gear I need anyways, and I've taken a renewed interest in crafting so I need all the storage space I could get.
Thought about just getting more retainers but I already feel the hassle with 5 retainers.