"Balance team tests the fights themseslves"

[Packetdancer]

...that it can be cleared with all the comps in regular player mode...

I feel safe in suggesting there is next to zero chance they actually try every possible comp. Just for the first seven, assuming no duplicate jobs, one pure healer/one barrier healer, and one of each role, you have (assuming my top-of-my-head math isn't wrong):

6 possible combinations of tanks (assuming no duplicates)
4 possible combinations of healers (assuming 1 pure healer and 1 barrier healer)
45 possible combinations of 3 DPS (1 out of 5 possible melee, 1 out of 3 possible casters, 1 out of 3 possible physical ranged)

For I think 1080 total possible party comps just for 7 of the 8 slots.

Whichever three DPS are picked, there are 8 possible DPS jobs remaining (4 out of the 5 melee, 2 out of the three caster, or two out of the three physical ranged). Now, not all of those possibilities apply, of course; MNK for your melee role and SAM for the extra is no different than SAM for the melee role and MNK for the extra. But even allowing for that, you have literally thousands of possible party compositions.

The chances that the team hand-plays through the entire tier at minimum ilevel with every single possible standard comp are breathtakingly unlikely, just given the sheer number of possible standard comps. I absolutely believe they did it back in ARR and, say, maybe Heavensward. But now? The chances are very, very slim that they actually have the time and manpower to play through the fight with every single party comp. Especially if there were any balance work to be done afterwards, when you'd then need to play it through on everything again. You almost certainly pick several 'representative' party compositions you assume give decent coverage, and judge the fight balance from those.

Which I suspect is precisely why this problem arose; those representative party compositions were probably insufficiently representative.

[TaleraRistain]

6 possible combinations of tanks (assuming no duplicates)
4 possible combinations of healers (assuming 1 pure healer and 1 barrier healer)
45 possible combinations of 3 DPS (1 out of 5 possible melee, 1 out of 3 possible casters, 1 out of 3 possible physical ranged)

Dating myself here

https://www.youtube.com/watch?v=w0i_ZFlGTVY

Use combinatorics, a faster way to count
Use combinatorics, you can multiply the amount
Use combinatorics, it's so easy to do
Use combinatorics when you need it
Let the numbers work for you

[SeiyaSoiya]

In one of the producer letters they explained how during testing they use an HP refill mode (refill rather than KO) to avoid having to wipe and restart. This intentionally enables the testing teams to go in blind but learn the fight much faster than regular players would. After they know the fight, the now-expert team confirms that it can be cleared with all the comps in regular player mode at min ilvl and without HP refill. I imagine Yoshi-p (busy as he is) would also use HP refill mode to learn the fights, and would probably clear them with experienced teams (though maybe still at min ilvl for a bit of fun.)

Obviously if they gave this sort of wipe invulnerability to players during progression it would make learning the fights much faster and easier. Players would also probably complain that the fights were too quick and easy to clear.

I think Yoshi-p also explained that the dev/testing teams are good players but not "world class" players, and they're always impressed at how well players optimize the fights (I imagine, without any evidence of course, that they're also grudgingly impressed with players finding and exploiting bugs and unintended behavior, even if it results in major headaches.)

that's interesting, is there any link to this liveletter?

[DixieBellOCE]

I am sure they do extensive testing and don't have any issues clearing the fights
however it really rubs me the wrong way that they pretend as if the dev/tester prog experience is the same as the player prog experience
you cannot and will never convince me that dsr was cleared during testing without the use of dev tools to checkpoint each phase and prog each independently

meanwhile actual players must always go back to phase 2 each wipe.
yes, you didn't need 3rd party tools to clear dsr, you had 1st party tools that are a million times more cheaty and powerful than any 3rd party tool

They also play on 0-1 ping on a localized server where clipping GCD's does not exist, therefore having slightly higher damage output than the regular player.

It's the same issue that arised in FFXI when Absolute Virtue was unkillable unless you used an illegitimate method that the dev team would abuse you for doing (And then patch it out of the game).
They tried to prove the playerbase wrong by showing the correct way of dealing with mechanics, while on 0-1 ping. THis caused an outrage and people clearly pointed out that in an ideal 1 ping setting, it was viable.

[Deveryn]

1st party cheats? lol. They have the people that built the damn fight telling them how it's supposed to go. They go through it to make sure it works.

[Bukachu]

1st party cheats? lol. They have the people that built the damn fight telling them how it's supposed to go. They go through it to make sure it works.

0 confirmation of this apparently so not sure what it all is based on.

[Deveryn]

0 confirmation of this apparently so not sure what it all is based on.

Based on the history of MMO development ...

[Valkyrie_Lenneth]

0 confirmation of this apparently so not sure what it all is based on.

Well, there's 0 confirmation that you have cleared or even entered them yet. Why should we believe you have any idea about balance?

[Packetdancer]

1st party cheats? lol. They have the people that built the damn fight telling them how it's supposed to go. They go through it to make sure it works.

It depends. Some companies will have a group of testers prog some content blind, though often that's not an in-house team. WildStar actually did some of that with the first Adventures and raids before the game was out; they had specifically chosen folks from the beta tester group run some bits of content blind and see if there were parts that were unclear. I don't think SQEX does that, admittedly.

And an in-house team almost certainly has a breakdown of the content's mechanics and progression, because you frequently don't have the time to let in-house testers prog something blind; they're in there to make sure the mechanics work as expected (which means they need to know what "as expected" is), and that the phase transition doesn't crash swapping arenas (which means they need to be able to get to said phase transition), and that a mechanic isn't so tight on timing that it's not possible for a party to move in time, etc.

The sort of testing and balance an in-house team needs to do almost requires they have the fight timeline there to work from.

And honestly, I would be shocked if the in-house team played through the entire tier with every possible standard party composition, especially if they did so every time the fight was tweaked during development; with the ever-growing number of jobs this game has, that would start to take a prohibitively long time to test every fight with every single party combination every time they tweaked the fight in question.

So, am I surprised they might've played with a party comp they were comfortable with, and just taken it for granted that it was doable with all party compositions? Not really; I can easily see that happening.

[AmiableApkallu]

So, am I surprised they might've played with a party comp they were comfortable with, and just taken it for granted that it was doable with all party compositions? Not really; I can easily see that happening.

As I put it in another thread:

5 choices for melee * 3 choices for physical ranged * 3 choices for magical ranged * 6 tank pairs * 6 healer pairs = 1620 possible seven-person party compositions with one player per job. Multiply that by… 4, I think, to get the number of eight-person party compositions? Anyway, I doubt SE is devoting the resources to exhaustively test that many combinations.

They're taking shortcuts, and it's not difficult to imagine how it might work to set the DPS check:

1. They start with the internal testing team's DPS with their chosen party compositions. They have some expectation for how players of some certain skill level playing those same compositions compare to the testing team, so they take the testing team's DPS and shift it up or down to get something appropriate for the players.

2. They then take into account that some party compositions will naturally perform worse than the ones they used in testing, so they shift the DPS down to accommodate those less-performant compositions.

3. They then take into account natural damage variance and shift the DPS down to cover some not-small range of that variance.

So, it doesn't matter how well SE understands job balance (2) and damage variance (3) if they start from a bad baseline (1) because they misjudged where the players would land relative to themselves, which can happen when their testing team's performance improves unexpectedly.