While it probably contributes to the amount of later dungeons with unskippable roadblocks, there's no reason not to skip them now.