Not even that 'our character doesn't remember how to wear a dress', Glamours in-game (in terms of lore etc.) are literally just an illusion magic. The Tank that looks like they're wearing a bikini, because they used Glamours to change their tanky defensive armour to look like a bikini, isn't wearing a bikini. They're wearing the tanky armour still, it just 'looks like a bikini' becuase of illusion magic. In terms of what else Glamours are used for, for example, we also make a Garlean Flag look like an Ala Mhigo Flag (Stormblood, the assault on the bridge in the Fringes), the Resistance disguises the pathway to Rhalgr's Reach as a cliff face (also Stormblood). So it's pretty free-form in what the illusions projected can be.
So, it's not even a lore explanation. It's an 'outside of the game' explanation for why we can't (and not a very good one IMO)