(I realize this is trivial, but still!)
Specifically for Dunesfolk, Piety is our highest starting stat. Removed, and we no longer have that "strength", whereas, for example, Highlanders with very low Piety, look even better with their "weakness" removed.
As a SMN, I would justify Duskwights having higher INT being offset by their lower Piety; their more powerful spells sustained by a weaker Aetherflow. As a Dunesfolk, I could eventually catch up with a Ruin III unavailable to Duskwights because I would recover more MP with my Aetherflow each minute.
Seems just a little unfair. ._.