I could see the ladder happening if it can do an item slot check. Like for each retainer you're gonna get the items of, you need 2 slots for. So each additional one adding another 2 required slots. Plus it can be a pain during busy hours, since I've had my retainers a bit slow loading in/leaving during the more populated times.