The thing is, retainers are retainers in the first place precisely because they are not cut out to be actual adventurers (this was stated outright when you hired one back in 1.0), but that they do excel in odd jobs that adventurers are meant to dislike:

Quote Originally Posted by FFXIV version 1.0 Gridania retainer vocate Gyles
"Retainers are men and women who offer their services in exchange for coin."
"Though they're seldom as sharp as adventurers like yourself, they're no strangers to life's more menial tasks. Tasks which your lot might not think worth your time."
Given my retainers keep bringing me back parts of houses that they're clearly stealing from construction sites (amongst other clearly 'hot' merchandise), I wouldn't trust them on the battlefield as much as I would trust an Ascian...