There's probably 2 major issues, one we clearly know and one we can only speculate on.

First, the clearest issue is the fact this game is built on the remains of 1.0. Both in regards to the underlying engine/code running the game and a lot of the design of the early 2.0 game retaining poorly fleshed out systems from 1.0 (i.e. classes, cross class abilities, leves). Given how fast a turn around they had from 1.0 to 2.0 they did a good job with what they had, but fundamentally the game needs a better engine designed for the game they want it to be, not a retrofit of an engine from a different game altogether.

Most discussions on the topic of how to fix or improve the game essentially boil down to them needing to just make a new engine from the ground up. Which of course by that point means they may as well just make a new game if they are putting forth that much effort.

Of course that's not to say all the blame can be left on the engine. There's clearly either a lack of resources to develop more imaginative content or lack of creativity to design it.

Diadem I think is a good example of this. It was referred to as exploratory missions, with the theme behind it being airships and exploration, yet it missed the mark on both those points. It had static monster spawns and only had the airship drop you off. On the most basic level to reach the goal of being an airship exploration content they could have made it that each island had randomized spawns. More so than that they could modulate it so each island could have an encounter module leaded onto it, something like a sky pirate encounter, dinosaur island, gathering node island, an island filled with traps and treasure, etc. With each encounter having a little variety in it in type/numbers of things spawned. That would give meaning to exploring as each island would be different each time you entered. As for airships, they simply could have made it so that to get from island to island you have to ride your airship. Even if all it is is players standing on the airship then telling an NPC to go to X island. It would be a start and could even open up allowing players to mount cannons while on the airship and have encounters with large flying enemies or other airships.

While I'm no programmer and more importantly I don't have the technical knowledge of their engines limitations (and neither does anyone outside the devs), I think those ideas are not a stretch to design and should reasonably work in the confines of how this game operates.

This is basically what I consider the second major issue; inability to implement creative content within their system, whether by a lack of creativity on their part or through a lack of resources to properly implement something more engaging. This point is more speculative as I've seen debates over whether the devs are creatively bankrupt or just lacking in the manpower to actually implement something actually new. Whether the formula exists because the devs feel safe with it or because its the only thing allowing them to churn out repainted content to keep the sub money coming in.

I have no idea on what reason it may be, I just know the end result is the game hasn't felt fresh in years. I hardly look forward to new patches and find myself on the game less and less outside raid nights. 4.3 has also been a bland patch and stretched way too thin, probably to maintain subs.

All things considered I get the feeling that they may never work on fixing fundamental flaws of this game. Because it would take too much to overhaul the entire engine and core of the game. Also, it's been 2 expansions and some 5 years without innovation and sticking to the formula and they can maintain enough subs and supplement lost sub income with the very over-reaching cash shop, so the higher ups know there's not much need to keep things fresh. I get the feeling after the SE survey we'll probably hear about the next FF MMO in about 2-4 years, while this game stays on autopilot to be low cost to maximize profit.