Here is your "Playerbase"

[SpankWustler]

So how exactly do they make content that is a challenge to people who have completely deconstructed every mechanic in the game, and then gone out and gotten all the best gear / jobs to beat the content, without completely breaking it for everyone else?

The "Reverse-Phalanx" effect possessed by the pickle-hating Tarutaru who is fought in the final limit break seems like a pretty ingenious equalizer. I'd fly to Japan using only my furious, flapping arms and poop on somebody's desk if every big monster had such a trait, but stuff like that is great for some monsters.

Basically, anything that makes horizontal improvement more desirable than vertical improvement for a specific fight helps to even things out between above-average and near-perfect players. It's a lot easier to grab a decent -Damage Taken% set or make a half-way decent set for several viable weaponskills for changing situations than to be the very best at smashing all that stands before you to atoms.

[saevel]

You're forgetting another problem with this idea: the time and effort required for the playerbase to figure this out through trial and error. There may have been some complicated yet elegant method for taking down AV before level caps increased, but it couldn't be found even with the "helpful" video the devs posted.

If you put your elegant system too far away from the path of least resistance, the playerbase would sooner dynamite their way through the mountain first.

No there was no "complicated yet elegant" method, AV was just designed to be unkillable. He didn't even have a drop pool or title, think on that before you comment again on AV. He was designed with the sole purpose to be the invincible brick wall that "high end" players would spend all their time trying to beat. It's a game mechanic for giving Munchkins something to do. It's also considered highly unethical.

SE just needs to do what EVERY OTHER MMO does and provide hints NPC dialog and monster feedback. We largely ignore NPC text in this game. Have a guy in the Tenshodo mention how "such and such NM is vulnerable to blah blah". .dat mining will discover it and soon enough people will figure out the system. SE spends too much effort hiding their poor work from the player base.