Well, being a warrior culture doesn't necessarily mean you'll be bigger and brawnier. Settling down still requires you to do a lot of physically intensive labor, building houses, working the land and what not. In contrast, Xaela probably eat relatively poorly and travel fairly light, considering their nomadic culture. Carrying capacity is kind of limited when you're on the road all the time.

That said, Au'ra girls might just be mooching off on the guys' hard work. Considering their intended moe appeal, it wouldn't be all that surprising, actually. They'd just go "Onii-chan!", the guys would have a nosebleed, and the girls would smirk as they rub their hands together. Hoho.