Thoughts on Feedback Questioning

Over on the Job Design Feedback thread, I updated the post with some new questions as I want to collect some more thoughts and ideas for when I do more feedback forms in the future, but I also wanted to bring up some talk about those questions here as it's something that we've discussed in regards to the healer feedback.

First, do you think I did a good, fair job at asking questions? Did any of the questions feel bias to you? Are there questions that you think I should rework, or questions I didn't ask that I should have?

Also, in regards to the 1 to 10 scale responses, what do you personally see as a "good" score, and what do you see as a "bad" score? What do those answers mean to you in regards to the job itself? At what score threshold should a designer really consider investigating changes to a job? Is a 7 out of 10 acceptable? Obviously, whether a job is good or bad and what kind of changes it needs to receive is going to involve a lot more than an average satisfaction rating, but would a low score be considered a red flag worth investigating?

None of the questions stuck out as being particularly biased nor ambiguous. I think the only other question I'd have been curious about would be a ranking of the role, although I can't see that being as easy to parse responses from or ask.

I'd say 6.5/10 is the breakpoint. I don't know how to put it into words properly, but 5/10 being the numerical midpoint and expecting all jobs to strive for 2 or more higher than that seems about right. Obviously not everyone can be pleased by every job, but jobs with under 6 should set off alarm bells, and 5 or less should be a cause for severe concern. If an entire role was consistent in score, that should be a good indicator for opinion too - or indeed if one role member significantly deviated from its fellows.
Since it's pretty clear that the worth of "1" isn't the same, with further distance from breakpoint becoming more heavily weighted, this should also be factored in. I imagine it's some kind of skew normal distribution, so higher numbers get better faster than lower.

Quote:

---

Originally Posted by ty_taurus

Also, in regards to the 1 to 10 scale responses, what do you personally see as a "good" score, and what do you see as a "bad" score? What do those answers mean to you in regards to the job itself? At what score threshold should a designer really consider investigating changes to a job? Is a 7 out of 10 acceptable? Obviously, whether a job is good or bad and what kind of changes it needs to receive is going to involve a lot more than an average satisfaction rating, but would a low score be considered a red flag worth investigating?

---

It's important to look at both "average" and "mode" to get a better grasp at what "score threshold" they should be looking for. Both of these values are important to look at to understand what the general consensus is for a given job. The 1-10 scale response makes a slight problem with the weight of 1. There is no existence of "0/10" in the score, which means people who dislike the job design entirely can only rate it at 1/10 at worst -- the rating moves what should be considered as "acceptable" up as a result.


10 = Great gameplay, great design. People put a 10 when they're satisfied with the job entirely. It doesn't mean the job is perfect, but there's nothing that detracts the score from being given a high rating.
9 = Good gameplay, enjoyable and good design. People put a 9 when they have few things to nitpick.
8 and above is where job design and gameplay is in a good spot. There is a solid idea and design choice that players like about the job. The job might not suit everyone, but the main opinion is positive.

7 is the cutoff point for job design being decent - it's slightly good with room for improvement. There are clear dislikes to the design, but a good portion of it is positive and the dislikes can be made better by next expansion at the very latest.

Between 6 and 7, job design is entering a very dangerous territory. It means people don't think the job design is actually good or doesn't have particularly positive impressions of the design. It just 'exists'. You can assume jobs at this score range is lacking something that dropped it below the cutoff point.

6 and under is not acceptable. Rather than looking at it from "a 6/10 means it looks average", it's "40% of the score is missing". 40% missing is a big red flag. There should be a full re-evaluation on job design to why nearly half of the score is missing for these jobs in terms of satisfaction.

Quote:

---

Originally Posted by ty_taurus

First, do you think I did a good, fair job at asking questions? … At what score threshold should a designer really consider investigating changes to a job?

---

My take: The questions with a 1–10 scale are unrelated to whether the design team should investigate changing jobs.

The questions ask, "how fun is this job to play?" and "how much do you enjoy this job?" I have more fun than not playing the healers, and would rate them on the higher half of the scale. However, I also think the design team needs to give them significant attention.

At the other end of the scale, I'd rate the DPS jobs as low as possible on the "fun" scale, with DNC and BLM being the two exceptions. Whether I think that means they need to be changed… eh, I actually have no opinion on that.

What one might reasonably (and correctly) conclude is that my enjoyment correlates more to the role than the job's design, and that "fun" is a function of far more things than just job design.

I tend to agree with pretty much all of the sentiments here. Job design is ultimately a massive, multivariate set of things that bleed into one another. Personal enjoyment, engagement, balance, aesthetic, all sorts of things go into it and each one should have its own criteria for red flags. Whether red flags in a specific category should trigger redesigns? Ehhhhh that's also subjective. The questions you asked were fair, and they were a decent swipe at getting the "enjoyment" portion down. For enjoyment, I'd echo what was previously said. Anything under a 6ish (or even a 7 if you're really strict) should be concerning. If you get one or two people who aren't keen on a job, that's one thing. If you get a massive tidal wave of feedback that says your design is 5/10, it's a pretty good indication that people find it boring. I wouldn't ever rate something I *liked* a 5/10, even though it looks "average".

Rather than asking how much players enjoy a job, do you think it would be better to ask how well designed they think it is?

Quote:

---

Originally Posted by ty_taurus

Rather than asking how much players enjoy a job, do you think it would be better to ask how well designed they think it is?

---

This would make it more subjective since it's impossible to define what "good design" is.

People could say Procs are awful design since it makes DPS rotations RNG based, despite DNC working just fine so it's not particularly a good question.

Mainly in the same boat as other people, the only real criticism I'd levy is that 0/10 should have been an option, so that 5/10 is the exact midpoint (as it is atm, 5.5 is actually the middle). I don't really see any bias, but maybe that's because if there is bias there it lines up with bias I have, who knows lmao

As for the ratings of things, to me I'd interpret things as:

10 - The job could get literally zero changes in the expansion and it'd still be great
9 - The job is teetering on perfection, just needs one or two tiny QOL changes to be perfect
8 - The job is very good and its great fun
7 - The job is good, and is fun enough to be a main without getting bored of it easily
6 - The job is fairly good , but there are some cracks that need addressing, can live with it till expansion but that issue better be fixed by X.0 or we riot
5 - The job is just 'okay', functions in content, but there are definite improvements that need to be made
4 and lower - Job needs serious help, it's pretty screwed and needs immediate attention (like, this can't wait for an expansion)

With the healers scoring 4's and 5's, they're (according to my interpretation) teetering between 'this shit need fixing ASAP' and 'they are barely treading water'. Which lines up with what I feel about them really, like WHM is 'okay I guess', it functions, it completes content. But it could be so much more. As for AST, or should I say, 4ST (heh), delaying the 'rework' to 7.0 has not done this job any favours. At least throw us some consolations like 'Lightspeed has 2 charges' jesus christ

Consider a test, like your SATs, or GCSEs, or whatever your tests from final year of higher education. A 60% 'correct answer' rate for these kinds of things is a pass, but only barely. If a job is scoring an average of 6/10, as others have noted, that's not really a good thing, that's a 'barely scraped it'. So, for example, the whole healer ROLE getting a 'fail grade' is pretty damning. Ideally we should aim for everything to be at least a 7. The melee seem to be pretty much there, there's improvements that can be made still, but overall it's a fairly good time to be a melee, it seems. Guess that's why they call it Meleewalker

In the perfect world the Ascians were trying to restore, every job has a 'satisfaction average' like BLM does, 8.5ish/10. Unfortunately, SMN and the healers exist

In regards to the scores I've received, I do believe it's important to take these values with a grain of salt because there's a limitation to the amount of people, and what type of people have responded. That said, there's also no way to tell what that error margin looks like either. If we did get a massive influx of responses from the larger player-base, would scores go up? Would they go down? Would they change by 1 whole point? 2 points? half a point? We have no way of knowing, but I do think it's worth mentioning for this discussion.

Regardless, I'll also share my own thoughts on the scoring system currently. With how it's organized now, any score I consider "low" would constitute of a thorough collection of the written response answers and use that as a method of identifying the core issue, and also looking specifically at what low-scoring reviewers complain about as well as also looking at what high-scoring reviewers are saying they like also. As a general rule of thumb, I'd look at a 7.5 average or higher as 'at or above a healthy range' and does not necessarily require something, though I'd also want to see that across all categories, not just the overall satisfaction. If a job were to have only a 7/10 score, that's not a 'bad' score, but I'd want to see if there are commonly mentioned QoL improvements that the job could use, though I most likely would think the job wouldn't require significant changes beyond that. It's when things start approaching 6 and lower that I'd look into potentially more notable changes, but again it's all case-by-case, which I think goes without saying for the everyone's answers here as well.

That said, I've also been doing some light research, and 0-10 over 1-10 is being described as better and to keep scales odd rather than even. I'm also seeing that 1-5 has the problem of accuracy. Values will naturally cluster around the 3-4 range, and it makes it more challenging to actually identify if there's a difference across multiple jobs, but there's also the 1-7 or 0-6 scale option also which is more a midpoint. Any preferences on 0-10 vs 0-6?

I also want to remove the "other" option from the "what does this job need more of/less of" questions and increase the amount of options to have more accuracy, and also make it easy to create a visual for that information next time around. For the healers, this is what I'm currently thinking:

What does (Job) need more of in future updates?

• Animation Updates
• Higher rDPS
• Stronger MP Economy
• Burst Healing Actions
• Regen Actions
• Barrier Actions
• Attack Actions
• Party Mitigation
• Party Utility (Expedient/Warden's Paean/etc.)
• Personal DPS Buffs
• Party DPS Buffs
• Crowd Control (Stun/Silence/etc.)
• Mobility Tools (Swiftcast/Icarus/etc.)
• Unsure/Nothing Specific

What does (Job) need less of in future updates?

• Total rDPS (This job is too strong)
• Total MP Economy (This job's MP management is too strong)
• Burst Healing Actions
• Regen Actions
• Barrier Actions
• Attack Actions
• Party Mitigation
• Party Utility (Expedient/Warden's Paean/etc.)
• Personal DPS Buffs
• Party DPS Buffs
• Crowd Control (Stun/Silence/etc.)
• Mobility Tools (Swiftcast/Icarus/etc.)
• N/A (Nothing needs removing)

This would be followed up with the question "Please expand upon your answers below, and are there any other things you believe (job) needs more of or less of?" And after that, a general "Share your overall thoughts on the state of (job) currently."

Do you think those lists are stronger? Are there any options for the more of/less of categories that you think don't need to be there, or options I didn't include that you think should be added to that?

To me the scaling looks this way:

10 = Chef’s kiss. Perfection. They can leave the job as it is & I’d consider it is okay.
8 - 9 = Great. Maybe there’s this few/little tidbit that they could’ve improve, but it’s still a great reception for me to keep playing it.
7 = Good. There are definitely few issues that sticks out like a sore thumb (an example that I could recall was GNB prior to its Bloodfest cooldown being increased to 120s), but still enjoyable.
6 = Okay. Same with ‘7’ but over here said job have more glaring issues (i.e. PLD prior to its 6.3 change where I personally still relish the gameplay for being different, but it was held back by their inferior defensives & damage output)
5 = Uh.. meh. Functional, but is not stimulating. Can be assured that the job will be dropped if they don’t receive proper care in the future. No jobs should be in this feedback range for longer than a single major patch.
4 and below = ’Why do you even exist’-tier. If played at all, it would be due to any reason(s) but gameplay being fun. I.e. playing X job only for their aesthetic, nothing else.

No comment elsewhere because I’m echo’ing what the other had already described.

Quote:

---

Originally Posted by ty_taurus

If we did get a massive influx of responses from the larger player-base, would scores go up? Would they go down? Would they change by 1 whole point? 2 points? half a point? We have no way of knowing, but I do think it's worth mentioning for this discussion.

---

If you really want to know, you could take a copy of the current results and archive it, and then start posting this topic in a lot of various places, eg Reddit, facebook groups (if people still use that), streamer chats while they're playing FFXIV (could be good to get eg Zepla + her chat's opinions, since she's on a break from the game atm). Then after a couple of weeks after putting it out in places where it'd gain traction, you check the results again and see what went up and what went down. For all we know, everyone here's super negative about SMN, but all the chat members of streamers and redditors will vote it 9/10, we dont know

Some minor news about the surveys on general is that someone finally selected "No rework" for AST. 130 people, and only 1 person thus far believes the job is in no need of change.

Quote:

---

Originally Posted by ty_taurus

Some minor news about the surveys on general is that someone finally selected "No rework" for AST. 130 people, and only 1 person thus far believes the job is in no need of change.

---

https://media.tenor.com/vb0xDbIpNdEA...ah-jameson.gif