If you don't want to be teleported there, walk to the dungeon entrance and then start the dungeon finder.