You don't need more than two retainers anyway. With two, you have more than enough space to have all your crafting materials, gear, and other items.