As mentioned before, our Blessing of Light helps a lot against the Ascian and Primal threat, but we're not all-powerful (we defeated Nidhogg with Hraesvelgr's eye, for example, which we them gave back); we're extraordinarily powerful adventurers, but there's also other very powerful individuals.
In the Nadaam, while we ARE more powerful than most people in there, we're getting a lot of help from the Mohl, Hien, Yda and Gosetsu, who are also very powerful people in their own right. It also helps that many of those tribes were also fighting each other; if they figured we were the biggest threat and focused us down, probably only Gosetsu's insane strenght and stamina would have prevented us dying on the spot.
As for the other one I think you may be talking about
While we are strong and gameplay =/= story, it's implied we didn't wipe the floor with Raubahn and it was a pretty even fight, especially considering he's handicapped; at this point, most of the "strongest men" in the alliance WANT to test themselves out against us, which is pretty much the point of Hullbreaker HM, as well
And we can't forget that
Zenos did wipe the floor against us TWICE, even if we ultimately defeated him at Ala Mhigo and as Shinryu,
it does serve to remember that, game mechanics aside, we're not invincible or immortal, we're just much stronger than the average but almost every group has someone close to our strenght.