Interesting Tank Research

There was an interesting write up posted on reddit looking at tank mitigation. We have a plethora of information on tank damage freely available and easy to see, but we often have nothing but theorycraft and 'feelings' when it comes to incoming damage and mitigation as it is considerably harder to casually parse out with all the variables that go into mitigation.

This is the 1st real stab I've seen at an in depth look at how we as tanks are mitigating damage. It's an interesting read with some surprising obvservations. I would highly suggest more people take a look.

Linky

With the never ending stream of balance complaints for years here, it's nice to see some numbers on the usually fuzzy defensive side of the equation. It's only 1 analysis and uses a specific set of parameters so I'm not going to claim this as some end all be all of tank defensive balance, but it is a far better place to start than what we often see. I just wish they used a less click-baity title lol.

No, I'm not summarizing it here. Go read it!

I'll summarize it.

"Unless all three tanks are played at optimal level, Warrior and Paladin are still superior for the average player."

Quote:

---

Originally Posted by Kabooa

I'll summarize it.

"Unless all three tanks are played at optimal level, Warrior and Paladin are still superior for the average player."

---

/sigh. Or you could read the article and see that it is actually a very nuanced set of observations. Some support the 'common sense' of this forum, but there is a great deal that do not and challenge the common beliefs and stereotypes of each tank.

But nah. Lets just post 1 liner and pretend it will magically capture a complex subject to get everyone riled up instead actually learning something based on data.

Quote:

---

Originally Posted by Izsha

/sigh. Or you could read the article and see that it is actually a very nuanced set of observations. Some support the 'common sense' of this forum, but there is a great deal that do not and challenge the common beliefs and stereotypes of each tank.

But nah. Lets just post 1 liner and pretend it will magically capture a complex subject to get everyone riled up instead actually learning something based on data.

---

The data does succinctly point that. Now if we're to address a brutally short summary:

* DRK is on the same level as PLD and WAR only at the top percentiles (This is expected.)
* Tanks in general don't know how to tank (shocking no-one)
* Raid shields are a thing for heals and the community (another non-shocker)

I think this paragraph puts it quite well:

Quote:

---

You're incredibly right, but all you've done is address the fact that the justification is wrong rather than the perception per se. "DRK" is fine. It's the class that's hard, and the players that suck. For people concerned about under-performing DRKs, it doesn't change much at all.

---

This just brings more data that the job does have an issue in design, on why its rather clunky and a rather jarring experience for casual players vs the other two who show better design overall. The floor was already high in HW, but that high floor was rewarding. Now its just unnecessary.

Quote:

---

Originally Posted by Izsha

/sigh. Or you could read the article and see that it is actually a very nuanced set of observations. Some support the 'common sense' of this forum, but there is a great deal that do not and challenge the common beliefs and stereotypes of each tank.

But nah. Lets just post 1 liner and pretend it will magically capture a complex subject to get everyone riled up instead actually learning something based on data.

---

I read it. It didn't say anything we didn't already know. In fact, it took a very pedastelly approach ("You guys aren't using your mitigation tools")

I'm not saying it's wrong. But it all ultimately means a whole lot of nothing. Damage in this game is real binary. Threats tend to come in small, pre-defined windows and everything outside of that is fluff damage that's almost entirely covered by HoTs which are never not going to be on your tank.

Mitigation takes into consideration of all 3 tanks at Lvl 70. One big problem for DRK at least was the mitigation problems it had without TBN. Anyone who runs any leveling content can agree

I'll have to read and share this with my tank friends. Thanks!

Quote:

---

Originally Posted by Derio

Mitigation takes into consideration of all 3 tanks at Lvl 70. One big problem for DRK at least was the mitigation problems it had without TBN. Anyone who runs any leveling content can agree

---

I'd argue War is now equally janky while leveling. Berserk has a really weird feel to it. Mitigation wise they're fine leveling it's just Berserk that's weird. Just my opinion from leveling up an alt War.

Quote:

---

Originally Posted by Mahrze

The data does succinctly point that. Now if we're to address a brutally short summary:

* DRK is on the same level as PLD and WAR only at the top percentiles (This is expected.)
* Tanks in general don't know how to tank (shocking no-one)
* Raid shields are a thing for heals and the community (another non-shocker)

I think this paragraph puts it quite well:



This just brings more data that the job does have an issue in design, on why its rather clunky and a rather jarring experience for casual players vs the other two who show better design overall. The floor was already high in HW, but that high floor was rewarding. Now its just unnecessary.

---

That's a bit of a cop out though. The post points out where (some) of the problems actually are and they aren't where people tended to think they were. Also, theres a bit of a jump from "Drk isn't being played well" to "Drk is to hard". Those aren't the same things. For example, one of the findings was that Drks do not use their keystone defensive actions (TBN and Dark Mind) as often as War/Pld use theirs. Drks tend to use TBN only for tank busters instead of as often as possible (according to the study). I don't think that is somehow a result of Drk being 'to hard to play'. Using TBN and Dark Mind more often is certainly within most players capability. The data can be used to point to areas that we are making bad decisions, such as "maybe I should try to use TBN more often". There is the general "People don't know how to tank" aspect present in all 3, but I think its a bit of a cop out to say "Well drk is just to hard and that's why people suck". I think part of it is because the public discourse discourages good habits and reinforces some bad habits. We can use this as a starting point to take a fresh look at how we personally are playing and how we are telling other people to play by using data to analyze play instead of 'common sense' that we have built up without good data to back it up. Right now there is a it of a negative feed back loop in some of these areas between using FF logs to justify a problem, but people are making poor decisions in play resulting in poor logs.

I see this study as a place to start over with a fresh perspective instead of hacking the old horses. I find this section to be the best take away.

Quote:

---

I hope that this study proves eye-opening to the public, as it has been influential to myself, and will be influential in my teaching style as far as my mentorship in The Balance.

---

Instead of trying to figure out how to make this stuff fit into the common camps to support our old positions, maybe its overdue to actually take a fresh look at this stuff.

Honestly not much shock. I had to get on to my co-tank for not popping Rampart enough. He complained about always dying in 6, I look at his logs he used Rampart twice.. Bro!! Lol

Quote:

---

Originally Posted by Izsha

That's a bit of a cop out though.

---

It isn't. The only thing that is not a "non sequitur" in that link is that we all should strive to improve. Which is a given but that comes with its own caveats (all grossly non-measurable):

* How fast the average player adapts to content and its challenges.
* How the meta strategy is spread and how it tends to exclude things. (see people vs skip soar)
* We still don't know the dimension of what is to come for DRK since all is known until last second.

The most important thing to take away from all that is to encourage people to "improve your performance". Otherwise, all those numbers only apply to "the top" and doesn't translate to the lower levels, where the most horrible practices are pervasive and the most misinformation is spread like butter.

And I think the average player would need these improvements and for us to make it easier for them to reach them.

that data only proves DRK have a lack of tools for being flexible, we still lack in physical mitigation options and some even suggest use TBN even for AA when is too risky to waste it, with the gear i have use it in boss dungeons is a waste it never pops, and on primals i have like a 50-50%, TBN is only effective against large aoes and TB and i was technically designe to be like that, if it was planeed to be used against AA the shield never will be so short.

but in resume we can survive and we can do it pretty well like the other 2 but in a clunky way and with less fluff mitigation and zero selfheals, in the end the problem is not the numbers we get at least on mitigation with are in some case lower, is how we get it and using TBN almost on coldown with the mechanics it have behind its not really good for poor-average players, i highly preffer the mitigation tools we have on heavensward that TBN, at least i will have the feeling i have something against fluff damage with dark dance and for physical with foresight.

Well of course using TBN more is going to increase DRKs defence, no reason to stop the presses for that revelation. But there’s a reason people don’t use it much outside of tankbusters.

What does a PLD use to put up Shelton? Oath gauge, something not used for much else than sheltron. What does it get for using sheltron? Take any 1 hit in 10 seconds and you gain a large chunk of mp and a free oGCD attack proc. There’s no reason to not use it and everything to gain for using it.

Now let’s compare that to TBN, you’re using mp, a precious resource to DRK used for pretty much everything else in their kit. What do you get? A shield that when broken gives you blood gauge, a not as precious resource as the mp used to get it. But that shield may or may not even be broken in the next 7 seconds depending on the kind of damage you take, the only guarantee it’ll break is a tankbuster.

Hence why it doesn’t get used much outside of busters, the design is just so ass backwards when compared to sheltron.

Sheltron uses a fairly expendable resource to give you mitigation that is all but gauranteed to give you a precious resource/dps increase.

TBN uses a precious resource to give you mitigation that MIGHT give you an expendable resource that will pretty much only make you break even on the dps you would have gotten spending the precious resource on dps.

Basically the risk vs the reward on TBN is out of balance except on tankbusters where you at least know it will break.

What the study doesn't get into is things like the usefulness of the mitigation. For example, when taking heavy damage at certain intervals (like tank busters or autos and aoes with mechanical damage) you will value that mitigation greatly, where as the mitigation from TBN is more smoothed out over the length of the fight. If you are using TBN nearly on cooldown, you will take on average the same amount of damage as the other tanks, thats what the data shows. Example, if a buster does 132k to a tank 10 TBN's doesn't equate to 1 hallowed ground use.

Whether or not that mitigation is keeping you alive more so than the other tanks was not studied/gathered. One thing which was painfully obvious was the extend to which tanks in general are not maximizing their defensive casts (not just dark knights this was clear on all jobs), which is also a conversation people should be having: mitigating effectively and not just for busters.

Its a good first step, but it certainly needs follow up.

Quote:

---

Originally Posted by Cabalabob

Basically the risk vs the reward on TBN is out of balance except on tankbusters where you at least know it will break.

---

I bet that if TBN cost half the MP and gave 50 blood immediately we'd see people using it a lot more.

Quote:

---

Originally Posted by Reinha

I bet that if TBN cost half the MP and gave 50 blood immediately we'd see people using it a lot more.

---

Miss read It i edit It.
The cost of 2400 converted to 50 Blood is to make It dps neutral, i dont think It Will help make out principal mitigation tool being used as a ofensive tool all the time, if is was a small mitigation tool like SAM thirt eye yeah but no with TBN

Quote:

---

Originally Posted by Kabooa

I'll summarize it.

"Unless all three tanks are played at optimal level, Warrior and Paladin are still superior for the average player."

---

Another summary:

DRKs have to push a lot more buttons to get the same output as PLD/WAR. Which is why there is a huge skill gap between the best and the average.

The constant struggle of players wanting to bring things down to their level vs. SE wanting players themselves to improve.

Quote:

---

Originally Posted by Canadane

The constant struggle of players wanting to bring things down to their level vs. SE wanting players themselves to improve.

---

If SE wanted players to improve then they wouldn't design 90% of their meat-and-potatoes content to be so easily accessible or render the rewards for completing savage content utterly irrelevant within a patch.

Furthermore, the response to "DRK has to work harder to get the same results" shouldn't be "git gud", the response should be a call for a better designed job. Machinist, for instance, is much harder to play than Bard, but is largely inferior and far more subject to factors outside of the players' control. Regardless of the fact that it's currently speedkill meta, I think it's safe to say that having to work twice as hard for the same result isn't good design. If DRK was slightly better than the other tanks in exchange for requiring more optimization and skill, then it'd be a fair trade.

Quote:

---

Originally Posted by Canadane

The constant struggle of players wanting to bring things down to their level vs. SE wanting players themselves to improve.

---

You right, lets return WAR to the 4.0 status.

Quote:

---

Originally Posted by shao32

You right, lets return WAR to the 4.0 status.

---

Honestly I liked 4.1 just fine.

Quote:

---

Originally Posted by Kabooa

I read it. It didn't say anything we didn't already know. In fact, it took a very pedastelly approach ("You guys aren't using your mitigation tools")

I'm not saying it's wrong. But it all ultimately means a whole lot of nothing. Damage in this game is real binary. Threats tend to come in small, pre-defined windows and everything outside of that is fluff damage that's almost entirely covered by HoTs which are never not going to be on your tank.

---

It said where the problems actually are, and how we approach helping people just supports it. How many times have we seen a need help tanking thread and the response is essentially "Mitigate busters enough so you don't die, then do as much damage as humanly possible". Rarely do we get the more correct answers about using mitigation on fluff, especially when healers are dealing with other things like preys or your co tank has an add. But we will turn around and complain about fluff damage mitigation as the cornerstone of many balance discussions. Our priorities are backwards. You can simplify the entire post to "git gud" but that's misses the point. Using rampart more often or TBN more often aren't a result of tanks being 'to hard'. A couple button presses a minute in a game like this is not skill cap issue. Its a knowledge and choice issue. If we aren't even sharing the correct knowledge and not encouraging a decent priority decision tree, then we obviously needed to be told what derps we are in this study.

Quote:

---

Originally Posted by Chrono_Rising

What the study doesn't get into is things like the usefulness of the mitigation. For example, when taking heavy damage at certain intervals (like tank busters or autos and aoes with mechanical damage) you will value that mitigation greatly, where as the mitigation from TBN is more smoothed out over the length of the fight. If you are using TBN nearly on cooldown, you will take on average the same amount of damage as the other tanks, thats what the data shows. Example, if a buster does 132k to a tank 10 TBN's doesn't equate to 1 hallowed ground use.

Whether or not that mitigation is keeping you alive more so than the other tanks was not studied/gathered. One thing which was painfully obvious was the extend to which tanks in general are not maximizing their defensive casts (not just dark knights this was clear on all jobs), which is also a conversation people should be having: mitigating effectively and not just for busters.

Its a good first step, but it certainly needs follow up.

---

Its a narrow study looking at a specific subset of situations. More definitely needs to be done, but I like where its goin. Anything that puts actual data analysis in front of our fluffy feelings I'm a fan. Stuff like this helps to inform why people 'feel' the things they do when making general statements, and in other cases remind us that our feelings are easily tricked and not to fall for them.

Quote:

---

Originally Posted by 347SPECTRE

Honestly I liked 4.1 just fine.

---

Me too. War was way more fun for me with multiple windows to work for and get a sense of payoff when executed. Just lame now. Effective, but lame.

That has been posted here a few times. The comments section does a fairly good job of dissecting some of the main problems with that post.

Outcome Measures
I think that if someone provides you a set of data, it's worthwhile to critically evaluate it for yourself, rather than accepting everything presented at face value. Is the primary variable being measured relevant to you? Is it something that would change your decision making? If it isn't, then the rest of the discussion has no value to you. GIGO.

Measuring total damage taken is a big red herring. Not all damage taken has equal importance. Are you mitigating a tank buster that would otherwise kill you, or are you mitigating a random auto that would have been soaked up by regens/fairy? A lot of healers won't even bother throwing out direct heals unless you need to meet a specific HP threshold to survive.

Inclusion/Exclusion Criteria
Next, look at the inclusion/exclusion criteria. The post states that it's looking at a subset of optimal play. Optimal play for whom? Was there a statistically significant difference in the combined tank dps in order to achieve that level of mitigation?

There are some situations in which using TBN will be optimal, for a given run. There are some situations in which it won’t be. The exclusion criteria vaguely states that "Excessive TBN usage will skew results" but doesn't clarify what measures were implemented to evaluate or prevent this.

If we're not really being strict about dps optimisation, it's worth noting that the exclusion criteria intrinsically excludes all usages of Unchained, despite it being a relatively minor dps loss.

It’s not clear what subset this is supposed to be representative of. Mitigation-light speedrun optimisation? Mitigation-heavy early clears?

The section on Living Dead states that "all damage for the duration is counted as mitigation". The poster has previously gone on record as stating that Living Dead is 10-20 seconds of invulnerability. Most of us know that this is factually incorrect.

It also states that initial data collection was collected from voluntary submissions, before switching over to random sampling. Were these initial samples done with a different mindset? Did any of the players involved know what the data was being submitted for? Or were they blinded to the process? If not, it skews the results.

Methodology
One of the difficulties in evaluating mitigation is that it's multiplicative. The damage that registered is only the apparent damage. To exclude the effect of party mitigation effects, you need to know what the true incoming damage is before buffs. This means looking at every active mitigation effect and working backwards to remove their individual effects. On every single hit.

The rule-set also doesn't clarify what happens to single target healer mitigation effects, like Adlo and Benison. How are these accounted for? If they are excluded, how so? If they aren't, then how does healer composition influence the results?

There is a lot of variation in how tanks split up responsibilities in a fight. Sometimes, the responsibility is roughly equal, with regular swaps. Sometimes, it's very one-sided. Does WD mean that the WAR was only active for add pick-ups, with the DRK holding the boss? Does it mean that they swapped after tank busters? Is WD one homogeneous population, or does it represent multiple, distinct populations, each of which divvy up the tanking responsibility differently?

Which then brings us to the last point. If your ANOVA shows that there isn't sufficient evidence to reject the null hypothesis, then what was the power calculation? Phrased differently, with only 10 cases looked at for each group, what was your likelihood of a type II error? And even if the data-set isn't underpowered, does failing to reject the null hypothesis make the null hypothesis true?

There's a reason why peer-reviewed journals specifically publish data which has statistical significance. Absence of evidence is not evidence of absence.

Wrap-up
It’s fantastic that people took the time and effort to gather this data, even if it’s for a parameter that generally doesn’t influence player decision-making (But who knows? Maybe you're interested in maximising your mitigation per second).

The main weakness is the write-up. The data presented is a red herring that doesn’t really have anything to do with the meme conclusions (i.e. DRK is fine as it is, git gud). The formatting of the essay, which would be commonly seen in scientific, peer-reviewed journal articles, is used here as rhetorical tool to give it undue authority and disguise the fact that it’s an opinion piece. The title is similarly sensationalistic, and likewise has nothing to do with the data collected. The only disappointment in all this was the read.

Do you know what really gets people to play a job? Show people how much you enjoy playing it. Watching people do eight WAR content clears back in ARR did a lot to popularise that job in the community. It showcases the job and what it’s capable of.

Likewise, when you see genuinely talented players, as in the case of the WF group for the first Ultimate, come out and say that they love playing DRK, it sends a message to the player-base. Not that DRK is free from problems, or that it’s perfectly balanced, but that it can be a lot of fun to take into difficult content.

There’s no ego there. No sense of superiority. Just a genuine love of the game.

We need more of that.

Quote:

---

Originally Posted by Lyth

snip

---

Yes, it has problems. Its a narrow study looking at a small subset of tanking. It uses a debatable set of assumptions and requirements (like the no unchained among other things). It uses data that is available (fflogs) which has limitations of it's own. It's not a peer reviewed article in a professional journal. But its the best thing we actually have to work with. That's why my title is "interesting research" not "Conclusive proof that X hypothesis is correct".

I'm not here to defend the scientific efficacy of a bit of number crunching about a videogame made by a few people in their spare time. But it is still the only attempt at any tank analysis that goes beyond theory craft and why it should get some visibility, props to the authors, and take a look at what we as the tank community can get out of it. We aren't going to get journal peer reviewed studies. Comparing this to rigorous professional academic standards is really a bit excessive. It was never supposed to be that. Take it for what it is, not for what it isn't.

As for "useful" mitigation on AAs etc, I think this is an excellent example of why we need things like this to remind us to reconsider. We constantly minimize the impact of fluff mitigation in statements like this that which encourage people to ignore fluff damage because regens cover it and only mitigate spikes, yet at the same time bemoan lack of fluff mitigation as a balance point (and point to FF logs damage taken to support the claim). This isn't a peer reviewed study, but it is certainly useful in looking at how we talk about tanking and frankly, we do need to review how were talking about it because there are constant inconsistencies in what we do and don't value. We cant both say that defensive abilities giving healers extra GCDs is a powerful benefit of good tanking, then in the same breath push fluff mitigation to the side, and that is just one example.

The study certainly has issues, and I also believe the focus on dark knight/the provocative tone is a weakness of the write up which takes attention away from what was actually found about tanking compositions.

However, I do want to point out a double standard here. If we want to demand peer review quality of work to justify a claim that we don't like, then we also need to require peer review of the statements being made that we do like. This is a process which takes time, as someone who submits work for peer review I would say a short turn around is 6 months to get through the refereeing process, and up to a year to see publication (depending on field). So if we want to hold our judgments of classes for half the life of an expansion before telling the developers and community what we think/critically assess so be it, but realize that applies to all criticism and claims, not just the ones we disagree with. To give an idea of time to produce just that easy statement on total damage taken the time to collect the data, remove party effects, and remove runs using tank stance, having weakness, tank deaths, and other such things which would skew results and then complete an analysis, interpret the results took nearly 60 to 80 hours of people's free time. If you want more nuanced statements about survivability more work and more in-depth work needs to be done, and I expect it will also take a large amount of time.

For a video game hobby, I doubt we are going to find people who will have the time/energy to submit research to a peer review process to justify multiple narrowly defined questions.

On the other hand, if we want to share ideas and improve people's methodology for teasing out these statements with actual justified numbers, then we need to talk and share as a community. Peer review process is going to slow that down and limit the information to a small subset of people for a long time (remember peer review requires the review be carried out by (a) qualified people (in this case requiring some advanced statistics knowledge) and (b) be impartial to the results), and isn't a road we should demand of people. That said, the paper has weaknesses, and one is that it only looks at total damage intake, and not buster/spike damage, which is more important for assessing how well tanks are surviving.

Quote:

---

Originally Posted by Izsha

/sigh. Or you could read the article and see that it is actually a very nuanced set of observations. Some support the 'common sense' of this forum, but there is a great deal that do not and challenge the common beliefs and stereotypes of each tank.

But nah. Lets just post 1 liner and pretend it will magically capture a complex subject to get everyone riled up instead actually learning something based on data.

---

No, it doesn't. That thread was trashed for that exact reason.

The post basically says, if you'd read it, that DRK performs only marginally worse than PLD and WAR at top end content and significantly worse than the other tanks for the majority of players.


It also compares cooldown usage amongst the tanks and reveals that in their "random" sampling (again, how can it really be random when he admits his bias in literally the first paragraph) that despite similar (lack of) cooldown usage, DRK takes more damage. It even says raid shielding is useless because they're used improperly.

More importantly it actually misses the true problems with Dark Knight. Dark Knight has no problem clearing content. The problems are (A) it has a lower baseline and (B) it plays like chocobo dung. DRK being played optimally still doesn't mitigate or deal as much damage as the other tanks and the post acknowledges this. And most DRKs arent even complaining about performance. They're complaining that the entire playstyle from HW, high APM, was removed. It's literally just spamming Dark Arts now.

TLDR - The post says "DRK has fewer tools than the other tanks, but this is okay, because the other tanks don't use their tools".

Which is obviously completely nonsense.

He blames the players for not using cooldowns, and then ignores that equivalent WAR/PLD are still performing at a higher level.

Check Bokchoy's post. Most aren't willing to repeat this same "debate".

Quote:

---

Originally Posted by Luin

Which is obviously completely nonsense.

He blames the players for not using cooldowns, and then ignores that equivalent WAR/PLD are still performing at a higher level.

---

I think that him saying 'DRK isn't as bad as you think it is, people are just mitigating poorly, and by the way most comps are mitigating awfully' is somehow detracting from his statement or anything. Saying 'In general people are playing poorly and if you played better you wouldn't be having this problem' isn't coming into your home and kicking your cat it's shrugging and saying the problem is yours, not mine. If his data shows that then neat. DRK might still not be as good as the others in Guardian but everyone knew that already so no surprise there.