Only real reasons to have 4 retainers are if you are an avid crafter, want to sell more items at a time or utilize ventures. If you do none of these things then the extras aren't worth it.

I have 4 retainers over 3 characters and just the miner ventures alone are worth it .... those shards pile up. I personally have multiple 4star crafts with the rest 3 star so I have mats all but filling up 2 of them lol. Up to your playstyle.