I acquired the climbing skill and the gloves KI from the quest in Marjami Ravine and proceeded to climb down one of the vines. My wyvern did not follow me down and I did not notice this before attacking something and died because of it.

I can only assume he decided to run all the way around which would end up taking a couple minutes for him to reach me.