Just have one potion that lets you change the DoW/DoM retainer to any other DoW/DoM you have leveled, and another potion that lets your DoL retainer change between any DoL you have leveled. It becomes a little unfair if you can switch them between combat and gathering.
