It was there in pre Abyssea FFXI and it's present now in FFXIV. They follow it because they don't see it as optimal to go to other areas now
This is having your hand forced. If I can do leves every reset that net me 10K SP or I have a choice to do other leves that net me 5K, who in their right mind will choose 5K over 10K? people only follow a pattern if its the best choice they have, give people better or more balanced choices and they will disperse. IMO leves shouldn't be so central anyway, they should be solo only for the person who doesn't have a lot of time on their hands to hop on the game and play a little, and there should be other options for party SP.

Perhaps if they made guildleves non-repeatable if failed, then people would have a tendency to do leves in more than one place simply to get their 8 leves worth of grinding out of every leve reset.
This could definitely help but I don't see people ditching Ul'Dah that quickly especially because of the Market Wards and since its convenient to level up around the city a lot of them wont leave.