Yeah, retainers (and their inventories) are summoned one at a time so the game isn't trying to load all of them at once. Even with their equipment and sale items on top of the inventory, that's still just a little over half of what the dresser shows at once.