Upon using a Retainer Fantasia I found this little error:
"You can now re-edit your retainer's appearance and personality. Remove all her gear and speak with a retainer vocate"

Both of my retainers are/have always been male.