Are Ultimates a major contributor to Job Design issues?

[Valence]

From their perspective, when they tried making non-raid content, they got attacked for it (Island Sanctuary etc), and concluded that they can't win whatever they do.

Not gonna lie, they have such a weird way to tease and speak about things they intend to do in liveletters, and when you get the final product in the next patch, it's nothing like what you imagined at all. Who would have thought we'd have to make weekly excel spreadsheets to keep our factories pumping in order to satisfy the archipelago stockmarket? Can anybody tell me with a serious face that they actually expected this out of it?

[Kisshu]

To be fair, they did say once that people need to reduce their expectations for Island Sanctuary. Though, they still refused to give us any info, so it wasn't clear how low we were supposed to go.

I recall they gave a few hints during the second fanfest leading up to EW, showing the artwork for IS and a few lines. Then it was more or less radio silence on it all the way till 6.1 live letter or such where they gave a picture of the island to show the scale. Even then during the 6.2 live letter we didn't really get any indication what the end product of each patch would end up with (Excel sheets stock market).

By not saying much, and only downplaying expectations in interviews, it created a foundation for players' expectations to grow, imagining Animal Crossing style with lots of activity. It was something like 8-12 months from initial reveal till we started getting actual information on it.

I would really like to know who the play-testers were for IS and what they reported to the developers.

[Valence]

The play testers are the devs themselves. That's how they test their flagship pve duties, why would it be different for side content like this? It's literally a self congratulating loop with no external feedback.

[Jeeqbit]

Not gonna lie, they have such a weird way to tease and speak about things they intend to do in liveletters, and when you get the final product in the next patch, it's nothing like what you imagined at all. Who would have thought we'd have to make weekly excel spreadsheets to keep our factories pumping in order to satisfy the archipelago stockmarket? Can anybody tell me with a serious face that they actually expected this out of it?

What I expected was what they originally said:

• A place to let your minions and/or mounts roam around (this turned out to be true).
• An island away from every other player (this also turned out to be true).
• Although they were unclear about everything else, there was a lot of speculation that it was intended to be a sort of life simulator. However, the implementation was thoroughly boring, probably because it's a tab-target MMO but I don't know.
• There was speculation we would be able to build things which seemed pretty certain considering an island is otherwise empty, which turned out to be true. However, as yet, there is no actual housing there.
• Given it was taking the Ishgard Restoration/Field Op slots, it seemed there would have to be some sort of progression system like those had, which explains why we got the ranks and rewards.

I think the reason it got managed by Mammets is because they quickly discovered that it risked becoming a grindy chore like MMORPGs used to be. In fact, on release day some people actually got to max rank by click gathering instead of relying on things like workshop EXP.

So the potential for it to be very grindy was there and they obviously nerfed that a lot by giving giant EXP awards from crafting, building and daily workshops.

Managing it with Mammets was obviously to consider players with only 10 minutes to play per week. I think the text that shows when you unlock them almost completely admits that. Knowing how concerned they are that everything works for extremely casual players, it was predictable it would be finished "quickly".

I mean, did it surprise me that even Yoshi-P got through the majority of ranks within the first livestream he did of it? Literally half finishing the content in under 2 hours? No, that didn't surprise me tbh.

The play testers are the devs themselves. That's how they test their flagship pve duties, why would it be different for side content like this? It's literally a self congratulating loop with no external feedback.

They have a QA team of over 100 people and also get staff that don't play the game to test it out (such as Marketing and PR teams from other departments). Their goal is literally that people who have never played the game before can get through it.

[Valence]

Why do they keep mentioning that they make their own inner circle of devs test their new content? Why does it even happen if they have a QA team of over 100 people?

[Kaurhz]

Why do they keep mentioning that they make their own inner circle of devs test their new content? Why does it even happen if they have a QA team of over 100 people?

What I wanna know is how do half of the issues in the game get through the woodworks if they have a dedicated QA team that large.

[Jeeqbit]

Why do they keep mentioning that they make their own inner circle of devs test their new content?

When they say that, it depends a lot on what type of content.

For casual content such as dungeons and alliance raids, they do mean all their staff, such as marketing and PR teams. They are looking for people who work for their company who are really not good and can't do a basic rotation.

If you mean high-end duties, obviously they get a static of really good players to test it, who test all their high-end duties regularly.

Why does it even happen if they have a QA team of over 100 people?

They are just testing for bugs. They find several thousand bugs and reduce them to a small number, shown in the "Known Issues" section of the patch notes.

What I wanna know is how do half of the issues in the game get through the woodworks if they have a dedicated QA team that large.

There are always bugs in software of any kind. So what all game developers do is have a priority system. They rate bugs depending on if they are a big deal or not. If they are minor, then they often put them to the back of the pile.

That's why the game always feels so bug-free. What you see as a "bug", is more likely a design decision, and it's important to distinguish this.

The game has a lot of bad designs and bad code and infrastructure, but that's different from an actual bug. Many things people want to say are "bugs" get put in the bug forum by players and SE move it to "Working as Intended".

[Kohashi]

What I wanna know is how do half of the issues in the game get through the woodworks if they have a dedicated QA team that large.

Very easy. You can never cover and test all scenarios even if you have 3 times that size. You cannot cover what essentially hundreds of thousands of players "test" at the same time and how each person tests the limit of their implementation.

Also, they also release patches with known bugs to stay on schedule and which they will address on the first hotfix. A lot of games do that. As long as there is no game-breaking bug, the patch is viable and playable. Who the F cares about untranslated text? It lacks polish, yes, but you can still complete the quest and thus the quest is still deemed functional.

[Kaurhz]

Very easy. You can never cover and test all scenarios even if you have 3 times that size. You cannot cover what essentially hundreds of thousands of players "test" at the same time and how each person tests the limit of their implementation.

Also, they also release patches with known bugs to stay on schedule and which they will address on the first hotfix. A lot of games do that. As long as there is no game-breaking bug, the patch is viable and playable. Who the F cares about untranslated text? It lacks polish, yes, but you can still complete the quest and thus the quest is still deemed functional.

I wouldn't expect them to catch nor trap everything, but I certainly would not expect to see some of the bugs that have been present in this games history with a team of 100+ people on QA. Especially when many of those bugs would be visible by just merely playing the game, and would not necessarily be a fairly situational bug, and no I am not talking about your "But it's only untranslated text" 'bug'.

Edit: and I absolutely would expect quite a lot of the bugs to be patched, especially for a game that deems bug exploitation to be against the ToS.

[Valence]

When they say that, it depends a lot on what type of content.

For casual content such as dungeons and alliance raids, they do mean all their staff, such as marketing and PR teams. They are looking for people who work for their company who are really not good and can't do a basic rotation.

If you mean high-end duties, obviously they get a static of really good players to test it, who test all their high-end duties regularly.
They are just testing for bugs. They find several thousand bugs and reduce them to a small number, shown in the "Known Issues" section of the patch notes.

Okay so let me get this straight, we were obviously talking about encounter design testing, and you replied to me with them having a QA team of over a hundred people. Then you proceed to tell me that actually this QA team mostly tracks bugs (so that's not encounter gameplay testing), that it's actually all their non QA teams testing those encounters on the side of their actual job, and then finally that the high end ones (like ultimates being talked about here) are indeed tested by a smaller circle of devs.

I'm sorry but this is all but a QA team, this is a ragtag team of various employees of whom QA isn't the core job and skillset they have, who do this on the side of their actual jobs, and there is even less of them testing the hardest difficulties. I don't have anything against this mind you, but then how does this invalidate what I initially said (and got from their liveletters/interviews when they talk about it)?