Interesting Tank Research

[Izsha]

I read it. It didn't say anything we didn't already know. In fact, it took a very pedastelly approach ("You guys aren't using your mitigation tools")

I'm not saying it's wrong. But it all ultimately means a whole lot of nothing. Damage in this game is real binary. Threats tend to come in small, pre-defined windows and everything outside of that is fluff damage that's almost entirely covered by HoTs which are never not going to be on your tank.

It said where the problems actually are, and how we approach helping people just supports it. How many times have we seen a need help tanking thread and the response is essentially "Mitigate busters enough so you don't die, then do as much damage as humanly possible". Rarely do we get the more correct answers about using mitigation on fluff, especially when healers are dealing with other things like preys or your co tank has an add. But we will turn around and complain about fluff damage mitigation as the cornerstone of many balance discussions. Our priorities are backwards. You can simplify the entire post to "git gud" but that's misses the point. Using rampart more often or TBN more often aren't a result of tanks being 'to hard'. A couple button presses a minute in a game like this is not skill cap issue. Its a knowledge and choice issue. If we aren't even sharing the correct knowledge and not encouraging a decent priority decision tree, then we obviously needed to be told what derps we are in this study.

What the study doesn't get into is things like the usefulness of the mitigation. For example, when taking heavy damage at certain intervals (like tank busters or autos and aoes with mechanical damage) you will value that mitigation greatly, where as the mitigation from TBN is more smoothed out over the length of the fight. If you are using TBN nearly on cooldown, you will take on average the same amount of damage as the other tanks, thats what the data shows. Example, if a buster does 132k to a tank 10 TBN's doesn't equate to 1 hallowed ground use.

Whether or not that mitigation is keeping you alive more so than the other tanks was not studied/gathered. One thing which was painfully obvious was the extend to which tanks in general are not maximizing their defensive casts (not just dark knights this was clear on all jobs), which is also a conversation people should be having: mitigating effectively and not just for busters.

Its a good first step, but it certainly needs follow up.

Its a narrow study looking at a specific subset of situations. More definitely needs to be done, but I like where its goin. Anything that puts actual data analysis in front of our fluffy feelings I'm a fan. Stuff like this helps to inform why people 'feel' the things they do when making general statements, and in other cases remind us that our feelings are easily tricked and not to fall for them.

Honestly I liked 4.1 just fine.

Me too. War was way more fun for me with multiple windows to work for and get a sense of payoff when executed. Just lame now. Effective, but lame.

[Lyth]

That has been posted here a few times. The comments section does a fairly good job of dissecting some of the main problems with that post.

Outcome Measures
I think that if someone provides you a set of data, it's worthwhile to critically evaluate it for yourself, rather than accepting everything presented at face value. Is the primary variable being measured relevant to you? Is it something that would change your decision making? If it isn't, then the rest of the discussion has no value to you. GIGO.

Measuring total damage taken is a big red herring. Not all damage taken has equal importance. Are you mitigating a tank buster that would otherwise kill you, or are you mitigating a random auto that would have been soaked up by regens/fairy? A lot of healers won't even bother throwing out direct heals unless you need to meet a specific HP threshold to survive.

Inclusion/Exclusion Criteria
Next, look at the inclusion/exclusion criteria. The post states that it's looking at a subset of optimal play. Optimal play for whom? Was there a statistically significant difference in the combined tank dps in order to achieve that level of mitigation?

There are some situations in which using TBN will be optimal, for a given run. There are some situations in which it won’t be. The exclusion criteria vaguely states that "Excessive TBN usage will skew results" but doesn't clarify what measures were implemented to evaluate or prevent this.

If we're not really being strict about dps optimisation, it's worth noting that the exclusion criteria intrinsically excludes all usages of Unchained, despite it being a relatively minor dps loss.

It’s not clear what subset this is supposed to be representative of. Mitigation-light speedrun optimisation? Mitigation-heavy early clears?

The section on Living Dead states that "all damage for the duration is counted as mitigation". The poster has previously gone on record as stating that Living Dead is 10-20 seconds of invulnerability. Most of us know that this is factually incorrect.

It also states that initial data collection was collected from voluntary submissions, before switching over to random sampling. Were these initial samples done with a different mindset? Did any of the players involved know what the data was being submitted for? Or were they blinded to the process? If not, it skews the results.

Methodology
One of the difficulties in evaluating mitigation is that it's multiplicative. The damage that registered is only the apparent damage. To exclude the effect of party mitigation effects, you need to know what the true incoming damage is before buffs. This means looking at every active mitigation effect and working backwards to remove their individual effects. On every single hit.

The rule-set also doesn't clarify what happens to single target healer mitigation effects, like Adlo and Benison. How are these accounted for? If they are excluded, how so? If they aren't, then how does healer composition influence the results?

There is a lot of variation in how tanks split up responsibilities in a fight. Sometimes, the responsibility is roughly equal, with regular swaps. Sometimes, it's very one-sided. Does WD mean that the WAR was only active for add pick-ups, with the DRK holding the boss? Does it mean that they swapped after tank busters? Is WD one homogeneous population, or does it represent multiple, distinct populations, each of which divvy up the tanking responsibility differently?

Which then brings us to the last point. If your ANOVA shows that there isn't sufficient evidence to reject the null hypothesis, then what was the power calculation? Phrased differently, with only 10 cases looked at for each group, what was your likelihood of a type II error? And even if the data-set isn't underpowered, does failing to reject the null hypothesis make the null hypothesis true?

There's a reason why peer-reviewed journals specifically publish data which has statistical significance. Absence of evidence is not evidence of absence.

Wrap-up
It’s fantastic that people took the time and effort to gather this data, even if it’s for a parameter that generally doesn’t influence player decision-making (But who knows? Maybe you're interested in maximising your mitigation per second).

The main weakness is the write-up. The data presented is a red herring that doesn’t really have anything to do with the meme conclusions (i.e. DRK is fine as it is, git gud). The formatting of the essay, which would be commonly seen in scientific, peer-reviewed journal articles, is used here as rhetorical tool to give it undue authority and disguise the fact that it’s an opinion piece. The title is similarly sensationalistic, and likewise has nothing to do with the data collected. The only disappointment in all this was the read.

Do you know what really gets people to play a job? Show people how much you enjoy playing it. Watching people do eight WAR content clears back in ARR did a lot to popularise that job in the community. It showcases the job and what it’s capable of.

Likewise, when you see genuinely talented players, as in the case of the WF group for the first Ultimate, come out and say that they love playing DRK, it sends a message to the player-base. Not that DRK is free from problems, or that it’s perfectly balanced, but that it can be a lot of fun to take into difficult content.

There’s no ego there. No sense of superiority. Just a genuine love of the game.

We need more of that.

[Izsha]

snip

Yes, it has problems. Its a narrow study looking at a small subset of tanking. It uses a debatable set of assumptions and requirements (like the no unchained among other things). It uses data that is available (fflogs) which has limitations of it's own. It's not a peer reviewed article in a professional journal. But its the best thing we actually have to work with. That's why my title is "interesting research" not "Conclusive proof that X hypothesis is correct".

I'm not here to defend the scientific efficacy of a bit of number crunching about a videogame made by a few people in their spare time. But it is still the only attempt at any tank analysis that goes beyond theory craft and why it should get some visibility, props to the authors, and take a look at what we as the tank community can get out of it. We aren't going to get journal peer reviewed studies. Comparing this to rigorous professional academic standards is really a bit excessive. It was never supposed to be that. Take it for what it is, not for what it isn't.

As for "useful" mitigation on AAs etc, I think this is an excellent example of why we need things like this to remind us to reconsider. We constantly minimize the impact of fluff mitigation in statements like this that which encourage people to ignore fluff damage because regens cover it and only mitigate spikes, yet at the same time bemoan lack of fluff mitigation as a balance point (and point to FF logs damage taken to support the claim). This isn't a peer reviewed study, but it is certainly useful in looking at how we talk about tanking and frankly, we do need to review how were talking about it because there are constant inconsistencies in what we do and don't value. We cant both say that defensive abilities giving healers extra GCDs is a powerful benefit of good tanking, then in the same breath push fluff mitigation to the side, and that is just one example.

[Chrono_Rising]

The study certainly has issues, and I also believe the focus on dark knight/the provocative tone is a weakness of the write up which takes attention away from what was actually found about tanking compositions.

However, I do want to point out a double standard here. If we want to demand peer review quality of work to justify a claim that we don't like, then we also need to require peer review of the statements being made that we do like. This is a process which takes time, as someone who submits work for peer review I would say a short turn around is 6 months to get through the refereeing process, and up to a year to see publication (depending on field). So if we want to hold our judgments of classes for half the life of an expansion before telling the developers and community what we think/critically assess so be it, but realize that applies to all criticism and claims, not just the ones we disagree with. To give an idea of time to produce just that easy statement on total damage taken the time to collect the data, remove party effects, and remove runs using tank stance, having weakness, tank deaths, and other such things which would skew results and then complete an analysis, interpret the results took nearly 60 to 80 hours of people's free time. If you want more nuanced statements about survivability more work and more in-depth work needs to be done, and I expect it will also take a large amount of time.

For a video game hobby, I doubt we are going to find people who will have the time/energy to submit research to a peer review process to justify multiple narrowly defined questions.

On the other hand, if we want to share ideas and improve people's methodology for teasing out these statements with actual justified numbers, then we need to talk and share as a community. Peer review process is going to slow that down and limit the information to a small subset of people for a long time (remember peer review requires the review be carried out by (a) qualified people (in this case requiring some advanced statistics knowledge) and (b) be impartial to the results), and isn't a road we should demand of people. That said, the paper has weaknesses, and one is that it only looks at total damage intake, and not buster/spike damage, which is more important for assessing how well tanks are surviving.

[Luin]

/sigh. Or you could read the article and see that it is actually a very nuanced set of observations. Some support the 'common sense' of this forum, but there is a great deal that do not and challenge the common beliefs and stereotypes of each tank.

But nah. Lets just post 1 liner and pretend it will magically capture a complex subject to get everyone riled up instead actually learning something based on data.

No, it doesn't. That thread was trashed for that exact reason.

The post basically says, if you'd read it, that DRK performs only marginally worse than PLD and WAR at top end content and significantly worse than the other tanks for the majority of players.


It also compares cooldown usage amongst the tanks and reveals that in their "random" sampling (again, how can it really be random when he admits his bias in literally the first paragraph) that despite similar (lack of) cooldown usage, DRK takes more damage. It even says raid shielding is useless because they're used improperly.

More importantly it actually misses the true problems with Dark Knight. Dark Knight has no problem clearing content. The problems are (A) it has a lower baseline and (B) it plays like chocobo dung. DRK being played optimally still doesn't mitigate or deal as much damage as the other tanks and the post acknowledges this. And most DRKs arent even complaining about performance. They're complaining that the entire playstyle from HW, high APM, was removed. It's literally just spamming Dark Arts now.

TLDR - The post says "DRK has fewer tools than the other tanks, but this is okay, because the other tanks don't use their tools".

Which is obviously completely nonsense.

He blames the players for not using cooldowns, and then ignores that equivalent WAR/PLD are still performing at a higher level.

Check Bokchoy's post. Most aren't willing to repeat this same "debate".

[DaulBan]

Which is obviously completely nonsense.

He blames the players for not using cooldowns, and then ignores that equivalent WAR/PLD are still performing at a higher level.

I think that him saying 'DRK isn't as bad as you think it is, people are just mitigating poorly, and by the way most comps are mitigating awfully' is somehow detracting from his statement or anything. Saying 'In general people are playing poorly and if you played better you wouldn't be having this problem' isn't coming into your home and kicking your cat it's shrugging and saying the problem is yours, not mine. If his data shows that then neat. DRK might still not be as good as the others in Guardian but everyone knew that already so no surprise there.