Quote Originally Posted by Ellyrion View Post
I sincerely doubt they do as much testing as you think. If they did, we wouldn't see half the issues with class changes every patch.
I think Yoshi described their testing system for jobs in two parts at max level, with one effectively being on a dummy and another in what was considered a "raid environment" to see how effective they were at maintaining performance under typical raid stresses and obstacles, leaving the majority of test results based on statistics drawn from those conclusions.

In the words of Mark Twain, "there are three kinds of lies: lies, damned lies and statistics". In this case, skills like Sustain were removed because the dev statistics showed it was rarely used at all, when if they had tested for something like open-area or solo gameplay, they would catch the importance of it that SMN players were well aware of before giving that ability the initial axe. There wasn't outrage because it was a popular spell being removed, but because there was no testing for the leveling process or circumstances such as solo content where support utility such as Sustain are more vital.

They're also drawing statistics from hired testers who may play the game differently (if they even play the finished product at all) from how the players may interpret a job to be played. Because of this, as long as there is little to no customer representation in the testing process to identify issues that dev testers may not pick up, then such issues will always occur. Even if there is any beta testing for players these days, there would seem to be an extreme under-representation of any data that could be yielded from it.