My two Gil worth since finishing the ninja quest line is that Doma is based off of China rather than Japan. (This could add some credence to the Dawn Farther and Dusk Mother parallelling with Vietnam's Creation Mythos)

So if it is indeed China then even after the Raen settled into one area the Mountains and River Valleys within them still make for a fairly harsh climate. So the Raen likely would still have the physical builds of the Xaela more from the environment than the life style.

As for their war based system, the Xaela seem to have specialization in their muscle builds. Males are taller and seem to be more naturally built for strength where as the females seem more naturally built towards speed. Both Exhibiting muscle builds that are primarily geared towards endurance. (A big thing for cultures that form in and around mountain ranges)

So larger bodies could be an accommodation to allow males to still have power with out risking endurance considering higher elevations have less oxygen. So the larger build with similar muscle mass would allow for larger lungs and a stronger heart to provide oxygen to the larger frame. The smaller Build with less hard muscle mass could allow the females a great deal more speed than their larger male kin for similar reasons, less muscle would make it easier to move quickly in lower oxygen environments without compromising the small stature that would be more preferable to a speed based fighter.

Lets keep using Roes and Lalas as examples. The Big Sea Wolf Pirate could easily crush a small Lalafell so underfoot uses the superior speed and smaller stature to aid his Rouge training and out maneuver the Pirate for the win.

It's the classic Muhammad Ali vs Spike Lee paradox, it'll only take one hit but you actually have to land it first.