To a point yes. Unless the quest specifically tells you to do a certain dungeon. Then you can skip that shit until later.
You'll be missing out on dungeon drops specialized weapons and armor if you skip them completely. And later the hard mode dungeons are they only way to gain tomes effectively.
But until level 50 shouldn't be an issue if you don't ggrind the dungeon chain all night. Just don't forget to do one a day for your extra bonus roulette.