The point that is often raised isn't that robes are inherently feminine so much as their design, quite often, leans towards feminine aesthetics. At least where healers are concerned.

I'd also note that there's often a strong push from those who benefit from the status quo to downplay such opinions.