It's possible their may be slight padded cap between player max and retainer max if under 50.

I cant remember exactly but im pretty sure, my BTN is 22 and my retainers max is 18 so leads me to think their may be 4-6 levels needed between retainer and player to get xp, the first venture always levels them up to level 3 so keep that in mind.
obviously we're in maint and it's just a suggestion but its food for the thought.

also as Hec has stated, you cant harvest things you haven't done so yet but that would stop you even attempting to send them off.