They're not going to.
The WoL gets one-upped by chumps during the main and sidequests all the time. You're as powerful as the story needs you to be. If you try to apply logic to it, it gets worse when you consider the New World is full of Lv90+ creatures.