Are you certain you didn't miss a quest giver? When you finish your third letter delivery there should be an NPC outside the leader's chamber/area who has the next stage of the quest. This is the one which then points you to Baderon.

I know I've missed talking to them before and then had to go back.