I believe Fran states you're being watched by them but you don't see them? Or was that FF12? I don't know, the fact of the matter is they don't attack if a Viera is with the party. The confrontation doesn't come from the lore, it comes from Yoshida specifically telling us that the blaze in the trailer is located in the sacred home of the Viera - he didn't need to share that information. Now it's certainly possible that something may happen where we may gain free passage, and a female Viera will escort us through the wood. And after that, we fully earn the blessing to wander freely without an escort. Of course, this is where the whole - well a confrontation seems likely, no? If they're being attacked then the Viera will respond in kind, their males first. The idea is that if a male Viera doesn't attack us, then we will come across one who is attacking the enemy.
The other side of this; people think that we won't see a male at all. And that the attack on the wood may not stir the males into action - a retcon of their lore. Or, that the males would simply act unseen in a fight against a shared enemy. Or, that all of the males have simply vanished. Since if there were NPC males then there is likely to be playable males. Without playable males then it's likely there will be no male NPCs. Or there will be male NPCs, and the release of playable male Viera is simply a matter of time due to armour limitations etc.
And this is simply uncalled for. Obviously, we didn't set fire to their homes, the point they're making is that the males who are known to be trigger happy will be even more aggressive to outsiders during a time of war. Example; Aura's were attacked by Ishgardians because the Ishgardians thought they were draconic - this hate came from the war.
Regarding the whole; they don't live in the village. Well, that hardly means anything. They don't live in groups, they don't interact with the community; but it doesn't mean they can't live close to the village and/or around it - they simply do not live in the hub. An add on would be Fran's words. 'They are watching us' meaning there's more than one in just that small area watching them.