On the issue of "risk", rather than looking to more difficult instances, it might be better for there to be more consquences for failure. It's far too easy to jump into an instance, jump out if you don't like the looks of it (or if it doesn't win on the first try), and jump in again with a new group. The ONLY risk in a failed run is the time you spent failing that run, and to cut down on that penalty players are far too ready to bail at the very HINT that things might not be progressing smoothly.

It might be worthwhile to set valuable loot behind instances that CAN'T be rerun at will. Demand some kind of payment item in order to participate, one that's difficult or grindy to obtain. If a player needs to fork over 1000 Soldiery for a Talisman of the Ancients, or it's a 5% drop rate from Boarskin maps, or whatever, you can bet that player's going to do their best to clear the run, even if the Internets claim the best setup is with a WHM BRD BLM PLD and you get a SCH healer or a MNK, or whatever else.

Players should be encouraged to want to overcome difficulties imposed by unfavorable party setups or inexperienced players, rather than turn tail and run when the going gets tough.

Unfortunate that the dread spectre of FFXI should appear right above this post (it's practically the Godwin's Law of the FFXIV forums >_<), because it IS true that FFXI abused this strategy to an absurd degree. Even mission storyline fights typically required a key item for entry that players had limited access to (usually could only obtain one key item per hour, though sometimes reacquisition was even harsher). Even so, used in moderation I think that "high-risk" battlefields requiring payment items for entry would give players a sense of danger that is much lacking in current content.