Exploring places, that you really aren't supposed to be in, is the true meaning of adventuring. As it stands now, nearly all of Eorzea is wheelchair accessible. The people against this should ponder it a bit further, instead of imagining a copy/paste of the video in op.

How about a system where the climbing/descending is just a means to get somewhere, and not an active control system that you have to navigate, since that seams the biggest complaint towards it. With proper item equipped, you approach the sweet spot, and get an action cue to climb/ascend/descend etc. Then, just a small simple 3 second scene loads showing your guy do the work automatically, and voila you reach random unreachable destination. What awaits, no one is sure till you get there! Something like omegas idea I'm pretty keen on too.

Think back to the floating island/teleport system in CoP zones in XI. But instead of a 3 second approach distortion and raise item scene, it's just you actually making an effort. Even though its all just an illusion. To other players you would just show up instantly.

See, fast and simple.

oh also, we need resident evil type ladder climbs back please