5.08 Patch Notes up

[Yandere-chan]

FFlogs counts raid buffs as part of the DPS of the job giving the buffs rather than the one receiving them now. In effect all jobs in that list are under the same circumstances.

That is an improvement, but since logs are anonymous and this data is from all logs posted, it is neither accurate enough to assess how the whole playerbase is doing nor how players of equal skill do against one another. E.g. If there were 200 Samurai logs posted and 100 Dragoon logs posted, and half of the Samurai logs were of poor optimization and only 30 of the Dragoon logs were of similar poor performance, it would make Samurai look worse than it is. Since there is no curation in FF logs for this by its anonymous upload system and these results are just taking in account all logs posted, that alone is more than enough to make the results unreliable.

As much of a pain it might be, you'd need to individually test each class with all other classes being the same in the test groups, and with the same players for the other classes present, and the players of each comparative class being compared against one another would need to be of similar skill (there would need to be a good way to assess this such as player's performance vs. theoretical max performance). You'd then, with this dedicated team, in both variations, do all end game content multiple times, and then take the avg. of those results. That is as close to accurate as we could get as players. Square likely has access to tools to get even more accurate results though, if they would take the time to test things on their end (it's possible that they have).

[ForteNightshade]

That is an improvement, but since logs are anonymous and this data is from all logs posted, it is neither accurate enough to assess how the whole playerbase is doing nor how players of equal skill do against one another. E.g. If there were 200 Samurai logs posted and 100 Dragoon logs posted, and half of the Samurai logs were of poor optimization and only 30 of the Dragoon logs were of similar poor performance, it would make Samurai look worse than it is. Since there is no curation in FF logs for this by its anonymous upload system and these results are just taking in account all logs posted, that alone is more than enough to make the results unreliable.

As much of a pain it might be, you'd need to individually test each class with all other classes being the same in the test groups, and with the same players for the other classes present, and the players of each comparative class being compared against one another would need to be of similar skill (there would need to be a good way to assess this such as player's performance vs. theoretical max performance). You'd then, with this dedicated team, in both variations, do all end game content multiple times, and then take the avg. of those results. That is as close to accurate as we could get as players. Square likely has access to tools to get even more accurate results though, if they would take the time to test things on their end (it's possible that they have).

That isn't how statistics work. Once you reach a large enough sample size, it doesn't matter. The average will more or less remain the same. Since we have several thousand uploads for every job, this threshold has already been met. What you see on FFlogs is roughly accurate representation of each jobs' output. rDPS is far from a perfect metric but remains efficient. Additionally, FFlogs allows you to partition based on percentile. Therefore, a 95% Dragoon will only be compared against a 95% Bard. And in every single instance, they will absolutely dominant them.

[Yandere-chan]

That isn't how statistics work. Once you reach a large enough sample size, it doesn't matter. The average will more or less remain the same. Since we have several thousand uploads for every job, this threshold has already been met. What you see on FFlogs is roughly accurate representation of each jobs' output. rDPS is far from a perfect metric but remains efficient. Additionally, FFlogs allows you to partition based on percentile. Therefore, a 95% Dragoon will only be compared against a 95% Bard. And in every single instance, they will absolutely dominant them.

Yes, that is how statistics work, unless you are throwing accuracy out the window. Anonymous unregulated logs with no control does not equate to reliable test data for class adjustments, not even to a remotely reliable extent. It is not a matter of sample size when the fundamental system is flawed. Any person at any time could purposely upload 100's of extremely poor logs of say, Dragoon, and drastically shift the avg. It's insanity to think a full average of logs from such a system acts as a basis for reliable class assessment. But no one even needs to do this. There can just naturally be a happenstance of an unproportionate size of terrible logs for various roles of those who choose to upload. Massing together averages from all logs from FFlogs will never be a reliable source for objective data on where classes compare against one another. To think otherwise is willful ignorance and a credence to the saying of there being lies, strong lies, and then statistics.

Even if you were going to use FFlogs to try to get an idea of where classes compare against one another, you'd need to spend a long time curating the logs selected, which probably shouldn't be done by just one person (or by a group of like-minded people) anyways due to the concern of personal (or group) bias affecting judgement during the curation process. Honestly though, you'd end up with a far more accurate result if you planned a test from the ground up as described in my original post. What was presented, however, was ignorant at best, lazy at worse, and purposely misleading at worst.

[ForteNightshade]

Yes, that is how statistics work, unless you are throwing accuracy out the window. Anonymous unregulated logs with no control does not equate to reliable test data for class adjustments, not even to a remotely reliable extent. It is not a matter of sample size when the fundamental system is flawed. Any person at any time could purposely upload 100's of extremely poor logs of say, Dragoon, and drastically shift the avg. It's insanity to think a full average of logs from such a system acts as a basis for reliable class assessment. But no one even needs to do this. There can just naturally be a happenstance of an unproportionate size of terrible logs for various roles of those who choose to upload. Massing together averages from all logs from FFlogs will never be a reliable source for objective data on where classes compare against one another. To think otherwise is willful ignorance and a credence to the saying of there being lies, strong lies, and then statistics.

Even if you were going to use FFlogs to try to get an idea of where classes compare against one another, you'd need to spend a long time curating the logs selected, which probably shouldn't be done by just one person (or by a group of like-minded people) anyways due to the concern of personal (or group) bias affecting judgement during the curation process. Honestly though, you'd end up with a far more accurate result if you planned a test from the ground up as described in my original post. What was presented, however, was ignorant at best, lazy at worse, and purposely misleading at worst.

... except they won't shift the average when Dragoon has over ten thousand uploads. A few hundred is throwing a water in a river. It would take several thousand to even move the needle, which is so ridiculously improbable it isn't worth discussing. What you're positing would require a massive coordinated effort to intentionally sabotage logs. Such an endeavor would be very easily noticeable and dealt with long before it did any damage. Put simply, the scenario you're describing just isn't going to happen. Now that doesn't make FFlogs the end all be all, but dismissing it outright or even implying the numbers aren't a decent estimate is just plain ignorance.

[Yandere-chan]

... except they won't shift the average when Dragoon has over ten thousand uploads. A few hundred is throwing a water in a river. It would take several thousand to even move the needle, which is so ridiculously improbable it isn't worth discussing. What you're positing would require a massive coordinated effort to intentionally sabotage logs. Such an endeavor would be very easily noticeable and dealt with long before it did any damage. Put simply, the scenario you're describing just isn't going to happen. Now that doesn't make FFlogs the end all be all, but dismissing it outright or even implying the numbers aren't a decent estimate is just plain ignorance.

Um, no. I said 100's. So, let's say 500. If you have 10,000 logs that avg at say 13k DPS, and you add even 500 logs of say 100 DPS, you will knock it down to around 12385 avg damage. But, as I said, this isn't the point, this doesn't need to be done. This just happens naturally by the nature of random uploads with no control in place. You literally have no basis to even determine if a log being submitted is way below what should theoretically be possible by competent play or not. It's literally just random logs by random people all averaged together. That is NOT accurate, at all. Not even a smidgen of accuracy exists in that. As I said, you could take the time to curate specific logs from the site, but the curation process would need to be very solid, but that even that would fall short of a proper test. To just average it all together though is just folly. It simply isn't a matter of sample size, as I said, the fundamental method that data is being collected by is flawed for the purpose it is being used for in that table.

Edit: It isn't actually as bad as I thought it was because the table is using the 95% percentile at least. I still think that isn't accurate enough, but that is much better than just all logs uploaded on the site like I had misread it as.

[Zyneste]

Um, no. I said 100's. So, let's say 500. If you have 10,000 logs that avg at say 13k DPS, and you add even 500 logs of say 100 DPS, you will knock it down to around 12385 avg damage. But, as I said, this isn't the point, this doesn't need to be done. This just happens naturally by the nature of random uploads with no control in place. You literally have no basis to even determine if a log being submitted is way below what should theoretically be possible by competent play or not. It's literally just random logs by random people all averaged together. That is NOT accurate, at all. Not even a smidgen of accuracy exists in that. As I said, you could take the time to curate specific logs from the site, but the curation process would need to be very solid, but that even that would fall short of a proper test. To just average it all together though is just folly. It simply isn't a matter of sample size, as I said, the fundamental method that data is being collected by is flawed for the purpose it is being used for in that table.

People performing poorly is what affects the percentiles, the skill floor. Deaths out of the control of the Dragoon are the only thing that would skew the accuracy and even then, those sort of deaths will affect all jobs.

If Dragoons have a difficult rotation during certain encounters (or are generally not understanding their class), it will be reflected in the percentiles and it will be easy to see the gap in the skill ceiling amongst players. "Job difficulty will affect performance" or something along those lines. This isn't really bias, this is what FFLOGs is meant to measure. How well each job does while doing mechanics. Dragoons as an example of course.

[Yandere-chan]

People performing poorly is what affects the percentiles, the skill floor. Deaths out of the control of the Dragoon are the only thing that would skew the accuracy and even then, those sort of deaths will affect all jobs.

If Dragoons have a difficult rotation during certain encounters, it will be reflected in the percentiles and it will be easy to see the gap in the skill ceiling amongst players. This isn't really bias, this is what FFLOGs is meant to measure. How well each job does while doing mechanics. Dragoons as an example of course.

I don't think you understand what I'm referring to. Someone made a table that takes every single log on FFLogs and just averages the rdps together to form a basis for how classes compare against one another. Percentiles on FFlogs isn't in the discussion as it has nothing to do with the table. Percentiles just show you how good you are doing relative to others of your class vs. everyone else by log info. Uploading a ton of poor results would just end up making everyone else look slightly better (considering how low 100 dps is, pretty much any even casual player would move up from such an influx). But this doesn't really matter with the point that just averaging all logs ever together on the site isn't a good way to determine where classes compare against one another. You'd be way better off curating which logs you are averaging to avoid absurdly bad logs that make no sense. Of course, an actual test that is very well organized and controlled is going to be more reliable than even that.

Edit: Wait, nvm. It isn't quite as bad as I thought. It is only 95% percentile. While I still don't think that's nearly accurate enough since there isn't a good way to determine if the 95%+ percentile's of each role is of equal standing to one another in what classes are capable of with what's on the site, it isn't as bad as just all logs being averaged. I would take the table results with a grain of salt at best. You'd be better off having a more detailed selection for which logs to compare than just the 95% across the board if you were to use the site's recorded logs.

[Vnolan]

Imagine if every run from the start of the tier were uploaded. Do you really think the percentiles would be what they are now?

How is gear taken into account with the logs?

Why are people talking about 95% percentile when that only affects by definition an extremely small percentage of skilled players who will have no problem changing to whichever job gives their group an advantage?

The top 10 speed kills for all floors have every DPS job with two exceptions: RDM is in groups that are a whopping 1 and 3 seconds behind 10th place for E2S and E4S. The horror.

[Yandere-chan]

Imagine if every run from the start of the tier were uploaded. Do you really think the percentiles would be what they are now?

How is gear taken into account with the logs?

Why are people talking about 95% percentile when that only affects by definition an extremely small percentage of skilled players who will have no problem changing to whichever job gives their group an advantage?

The top 10 speed kills for all floors have every DPS job with two exceptions: RDM is in groups that are a whopping 1 and 3 seconds behind 10th place for E2S and E4S. The horror.

I agree, if every run ever was automatically uploaded, the data would be different. FFlogs is pretty loose though in what is allowed. I really just don't think FFlogs is that accurate when determining exactly how classes compare against one another. It's way too big of an assumption that the 95% percentile is of equal skill to one another from an aggregate of random users from a small subset of the entire playerbase. It's entirely possible and even likely that many of the best statics with the best players don't even care to upload to FFlogs, and thus aren't being accounted for. Just as it's possible that some roles have more below avg. players than others pushing worse players into higher percentiles vs. other roles. It's just better to take that kind of table with a grain of salt rather than treating it as some sort of defacto assessment of where the classes are against one another. I can only imagine that Square has there own means of testing how classes are performing which are way more accurate than what a quickly put together table from FFlogs can offer.

[Barraind]

You literally have no basis to even determine if a log being submitted is way below what should theoretically be possible by competent play or not.

Except you do.

Logs taken into account for dps numbers are KILLS, which gives you multiple easy confirmation points. Non-kills are kept off those breakdowns and attached to the users kill submission (if any) for reference by the submitter and submitters group.

You have to submit the logs as is, tied to an specific account uploading it. If someone is posting substantially different parses than everyone else (and if you're moving the damage off of one class, it has to be moved ON to someone else, which would also make THOSE numbers substantially different), the damage would still have to work with the kill time mentioned, AND every one of those bosses enrages, so you have a minimum raid damage threshold you have to meet.

Additionally, it would be seen as instantly suspicious when you have a party with someone doing that poorly over that many kills.


The amount of effort you would have to put in to altering logs in a convincing way to boost or tank something even a little bit would be considerable.