Did you have additional retainers at one time then delete some?

If so, it could be the ones you deleted were the retainers that were obtained with the base game while the ones you kept were the retainers tied to the additional fee.