I know equipment doesn't have an effect on that but is it the same if you unequip all gear on your retainer? Maybe there's some back end modifiers on gear that doesn't effect player characters