This is definitely how glamours should work and would solve not just your complaint but many others.
Though I'll point out that robes are not a female-only thing. Robes are a pretty classic magic-user clothing item across a range of subgenres and worn by males and females, so I do disagree that WHM gear having robes means it was designed "for women".
