Something the FFXIV developers should learn from the WoW developers

[Kazrah]

Blizzard doesn't make changes to content based on tester feedback once development moves out of alpha and into beta.

Except no gaming company tests for feedback; they test for analytics. Numbers and percentages related to content being done are much easier to read than any (mostly) jibberish that winds up in player feedback, which is why any written feedback is usually taken with a grain of salt. Looking back nowadays of when I took part in the ARR beta, I'm fairly convinced that whatever reasons SE may have had for player testing they believe can be done through in-house testers and mathematicians.

Ironically though, what testing possibilities that could be offered to players wouldn't involve story content at all. The same means that have been stated for testing job gameplay for example (target dummy testing and fight mechanic simulation testing) could be done by players willing to do it at no expense to the company.

[Valkyrie_Lenneth]

Except no gaming company tests for feedback; they test for analytics. Numbers and percentages related to content being done are much easier to read than any (mostly) jibberish that winds up in player feedback, which is why any written feedback is usually taken with a grain of salt. Looking back nowadays of when I took part in the ARR beta, I'm fairly convinced that whatever reasons SE may have had for player testing they believe can be done through in-house testers and mathematicians.

Ironically though, what testing possibilities that could be offered to players wouldn't involve story content at all. The same means that have been stated for testing job gameplay for example (target dummy testing and fight mechanic simulation testing) could be done by players willing to do it at no expense to the company.

I don't know that i trust players opinions on jobs after the SAM speculation before launch/ and the numbers people saw on some skills at launch. Thats one of the reasons its in such a poor place right now, everyone complained it was too strong because they saw potencies, but it wasn't, and after like, 4 weeks, other jobs got buffed due to the backlash.

[Kazrah]

I don't know that i trust players opinions on jobs after the SAM speculation before launch/ and the numbers people saw on some skills at launch. Thats one of the reasons its in such a poor place right now, everyone complained it was too strong because they saw potencies, but it wasn't, and after like, 4 weeks, other jobs got buffed due to the backlash.

And that backlash (along with the dumpster fire that was the tank accessory debacle) could've been avoided by having players test SAM beforehand and realizing that the numbers are intended to offset its complete lack of utility (which I'd argue is why it's in a poor place these days).

[Berethos]

Except no gaming company tests for feedback; they test for analytics. Numbers and percentages related to content being done are much easier to read than any (mostly) jibberish that winds up in player feedback, which is why any written feedback is usually taken with a grain of salt.

A good gaming company will look to both, and Blizzard does do this - but here's the catch: Most of the written feedback they've looked at and implemented is related to class design/balance. Does a change make something feel clunky and weird to use? Does the payoff for a particular rotation feel lackluster? That's the kind of written feedback they look for. Occasionally that written feedback will include whether or a particular piece of content feels rewarding (lately it's been leveling speed and scaling). There's plenty of "this isn't that fun" feedback, of course, and it's rare that those things get changed, leading to an all too common refrain - "Why wasn't this caught in beta?" "It was, Blizz just didn't listen again."

For the analytics - during the beta, it's about stability, and balance, and things working the way they are supposed to, NOT numbers related to content simply being done. I've been a beta tester in the betas for 5 out of their 7 expansions (including the latest one), and we generally just play through the quests and check out the classes/dungeons, or test what they specifically tell us needs testing.

The analytics for people actually doing the content (and presumably having fun with it)? That's the kind of thing that would get tracked in the live environment and is data that would be applied toward the next round of design decisions...and even then Blizzard has a tendency to decide to do what they think is fun in the end anyway.

Analytics for people doing the content should be taken with a grain of salt anyway - if you hate it but it's the most efficient way to get something, that's what you end up doing. Path of least resistance and all that, right? From just a numbers perspective, that "dislike but doing it anyway" element would never show up and instead it would look like the players at least like the content because they keep doing it.