OK, so this might be overkill, but whatever.

On my main character, I have 10 retainers, 4 combat, 3 miners and gatherers.

I also have a few alts with all of their retainers dedicated to a single role.