I suppose I offered a problem without a solution.

Make a raw material crafting bank.

Or, additional two retainers are unlocked with each major content block.

Completing ARR gives two more retainers, completing Heavensward gives two more retainers.