The only way to improve success with difficult mechanics is through practice. Unfortunately, players often find it difficult to get into groups to experience truly difficult content - i.e., endgame raids or trials - giving them minimal exposure to hard mechanics that would punish them for subpar play. This is pretty easy to pin down why: weekly lockouts and player frustration. Early on, most groups are going to look for players that are already experts at their class, can handle difficult mechanics, and favor people with backgrounds in raiding because it's the most efficient way to go about obtaining loot with weekly lockouts. For new players who have not mastered the game, they are typically unwanted in endgame content until they can get limited experience through learning parties, extreme primals, what have you. In addition, endgame is missing a lot of what I would call "midcore" tier content, which could be vastly improved by installing a new type of dungeon.
By giving more varied forms of challenging content, especially content that is not gated by a weekly lockout mechanic - dungeons, I think, would be a great alternative to raids.
I gave a couple of other examples other than tomes of what could be rewarded in challenging dungeons, but I guess you missed them. I think people would be willing to participate for a chance at decent gear drops, glamour drops, unique minions, or even titles from specific instances. The only incentive for clearing the Second Coil of Bahamut (Savage), after all, is the title.
And while providing a DPS measure may not be the best way to improve player performance, it can have a "glass shattering" effect for players who are not aware of how suboptimally they are doing. I mentioned mitigation percentages for tanks, overheal for healers, or a general training hall. The idea is not to make it easier for low DPS players to be called out, but for people to be able to take action to improve themselves, which is why I suggest only showing personal DPS/mitigation/overheal compared to an average by class, versus displaying stats of the whole party to the whole party. If you can think of a better alternative, I'd love to hear it.