I think the current system can definitely be improved to work better and create better incentives for players wanting a challenge to remove the advantage high level gear gives by setting the ilvl of all gear to the base level of dungeon, a feature that already exists in the game. A gambit system could also create a higher challenge level for existing content in a reasonable amount of work.

I don't think it needs replacing. Both horizontal progression models and vertical progression models have pros and cons. Overhauling FF14 to be a horizontal progression game would, I suspect be a huge amount of work, and only replace one set of issues with game progression with a different set. The investment in resources to facilitate such a change almost certainly have a detrimental effect on content release and that would have a worse effect on player happiness than the issues with progression do.