Reading through this, I am wondering...

Most of the time when we run into him, he's out and about, mainly dealing with us (and whatever pixies decide to play), but when we finally defeat him, it's in Eulmore, where Vauthry had just used his Sin Eater control thing. Now, he says that he isn't enthralled by Vauthry, but that doesn't mean he wasn't affected by what Vauthry did. He could have been weakened because Vauthry accidentally messed him up without meaning to. It'd give some sort of explanation as to why we managed to beat him, and still allow the fact that on normal conditions he could beat us to a bloody pulp.