The thing is, where are they going to get that concept of gender from, and form that identity upon, if the only thing they know about men is that they're not allowed to live in the village?
And apparently the women do everything that the men would do anyway, besides the additional ultra-survivalist skills that need to be taught from warder to apprentice.
At most you'd have some kids showing a preference about what role they'll have to take up, but that isn't necessarily tied to gender even in a culture with the normal concepts of male and female.
I think I'm missing the subtext, but you've got it upside-down saying they do it to "prove their masculinity". Their masculinity has been inarguably proven by the fact that they grew into a man, and that is why they have to get out there.
I don't think the idea is supposed to be easy to swallow. It's meant to be a harsh strict societal rule based on tradition over practicality. That's the only way a system like that can stay intact. If you start making an exception for the clumsy man who is good with kids and the woman who wants to be a warder, pretty soon it stops being a gender-based system at all.
The big thing is, tradition and religion make people throughout history do a lot of things that make no sense from a secular perspective. Do this, wear that, eat certain foods, don't approach the sacred mountain or the wrath of the gods will fall upon the village and it will be all your fault.
And this being fantasy fiction, there's a particularly high chance that the gods actually are ready to inflict their wrath.
If it's a matter of practicality, determining societal role purely by physical gender isn't a great approach, and maybe it would be helpful to let the warders come home for dinner too.
If it's the sacred will of the forest that men become warders and leave the village, you don't argue with the sacred will of the forest. That's just the Way Things Are and you take up your role or accept exile as your only other option.
Also, at a meta level, the whole system was concocted as an explanation for why there are no male (or male-model) Viera in the village. It wouldn't really make sense if it didn't strictly ban men from the village.