You are right that in Yotsuyu's specific case each individual Doman shouldn't be considered guilty as such. However there are two important points regarding Yotsuyu's story.
The first is that what happened to Yotsuyu was legally done. Those with power allowed it. She was sold at a young age as a bride to an abusive man and when he died sold again to a brothel. Regardless of the reasons of why it was done, it was allowed. At the very least that makes Hein's father partially culpable. Add to that that assuming her comments are right, her pleas for help were met with people telling her to put up with it. That suggests a culture that looks the other way. This isn't that far fetched considering how we see the Domans and even the Ala Mhigans acting under Garlean rule. However remember the first time we see Yotsuyu? She gives a man a gun and tells him to kill his neighbours and then his parents. I think that was intentional. Its easy to accept what is necessary to survive when it is someone else having to pay the price. Its much harder when its things you care about.
The Second, and this is connected to the first, is that Yotsuyu is almost certainly not a unique case. She was just placed in a position where her story came to light. In a society where a woman can be forced into prostitution or sold off like a slave you honestly think she was the only person it happened to? If she hadn't been given soldiers we probably would have never found out. Yotsuyu is a indicator of potentially the true sin of Doma, the willingness to accept injustice for the sake of peace and safety.
Part of the reason I suspect this is the direction it is going is Goesetsu. We know he did slaughtered people for the Garleans at the orders of his Lord. Doing so placated the Garleans and helped keep Doma safe but if you came at this whole situation from the direction of Goesetsu's victims you would probably see the samurai in a very different light right now.