Just quietly curse your retainer's name under your breathe after they give you another damn fish/furniture/entire house before you send them off on another quick exploration again.*

Within the next few ventures they'll feel bad for disappointing you time and time again and bring you some of that new rare dye or some reasonably/extremely valuable crafting item they mugged somebody for. Works 99% of the time for me.

*For extreme disappointment, send them off on an 18h exploration when you don't even need to log off yet. That'll show 'em for wasting your time.