The thing is, the whole idea around retainers came about because of how in FFXI players purchased extra characters specifically for extra storage space (called 'mules'), as you could freely move gear and items between different characters there, but each extra character after your first was an added fee to your subscription. SE obviously saw the potential in this for FFXIV and so retainers are the result.

Story wise it was actually stated back in 1.0 that retainers are not adventurer material but are better at handling mundane tasks that adventurers generally considered beneath them like item storage and buying and selling stuff at the market, and this is why retainer ventures they're shown.. being.. less than satisfactory in a more 'adventurous' role (my maids are maids for a reason, they may have combat experience but party members.... they are not.).