The majority of players are not interest on this. They just want to run as flash from one end of the dungeon to the other to get their cookies (experience and tomes). And to do this, they want dungeons to be as linear as possible. To the point that a lot of players do not even know the story, and most only know the spoken parts of it.

How many of you stop to read the notes in the dungeons? And how many of you did it while having other people in the party? How many do the Temple of Quarn puzzles opening all rooms? How many do the full Hakuke mansion?

Square tried to give better dungeons, but players didn't care in the past. The focus of players is in getting the cookies as fast as possible. So why should Square do a non linear dungeon?