Treasure Hunter Testing

[Urteil]

Our development team just figured out that 10 > 9.


It is only uphill from here folks, there is nothing they can't conquer.

No curelock they can't fix.

No menu's they can't make unterribad.

[Ophannus]

Test what "Enhances Soul Jump" effect does on Lancer Cuissots +2. Nobody has any idea what these do at all.

[Motenten]

For a sample size of 500 mobs, and assuming the drop rate of pugil scales is somewhere in the neighborhood of 20% (Wiki lists 16% for Sicklemoon Jagils in the same zone, though with a pitiful sample size; not really relevant, just need to know roughly where on the continuum the drop rate is likely to be), the 95% margin of error for that test is 3.5%.

That means that if the drop rate of the two runs differed by less than 3.5% then the test proves, literally, nothing. It is not statistically significant. If the drop rates differed by more than 3.5% but less than 7% there's still a chance that the two populations have identical 'real' rates, and the results are less than trustworthy. If the drop rates differed by more than 7% I'll be extremely surprised.

That said...

2. TH9 vs. TH10
Since it would take a ridiculous amount of time to test this by manually increasing the Treasure Hunter effect to 9 or 10 via attacks and what not

Which basically means you guys are admitting that putting more than TH3 on a mob is useless, which basically means TH+ gear is useless. Give me a reason to bring a thf. (at least you admit TH9+ is useless, so why even implement it, make it cap at TH10).

Camate's statement does not have anything to do with your interpreted meaning. He's saying that if you want to sample 500 kills with TH 9 and TH 10 the old fashioned way, you have to get enough procs to reach those two tiers under normal fighting conditions. That means you'll have excess samples of TH 6/7/8, and maybe 11/12, and have to fight in such a way that the battles will last long enough to get said procs.

Since they were only interested in testing TH9 vs TH10, that's a huge amount of wasted time and effort. Of course if the players want to test those values they have to expend that effort, but the devs can skip past all that tediousness.

None of which has anything whatsoever to do with the usefulness or uselessness of any TH tier.

[Motenten]

And you seem to fail to understand the meaning of statistical sample sizes.

For example, suppose that TH9 gave a 20% drop rate on an item, and TH10 gave a 25% drop rate (pretty substantial for only an extra +1 proc when you're already starting at 9). Would you consider that a significant increase in drop rate, implying a significant value in going from TH9 to TH10? I think most people would.

However, 500 kills for each TH level is not enough to statistically separate those two values. 1000 kills is not enough. You'd need about 4000 kills to statistically distinguish a full 5% increase in drop rate (a 25% increase in drops received), and even then all you know is that TH10 > TH9; you still couldn't really be certain of how much of a difference there is.

Your complaint that you need thousands of kills to be sure of a difference between the two TH levels is merely a reflection of someone who doesn't understand statistics. You will *always* need thousands of samples to get any kind of decent understanding of the underlying rates of anything in a game like this. You can only get away from it if you're dealing with data that you know is discreet and precisely repeatable, such as exact haste or enmity values.



Now, to take things further aside:

I've read one bit of testing which implied that TH1 was a 50% increase in drop rate over baseline, TH2 was an additional 50%, and each additional point of TH above that was +10% (a fairly reasonable model, really; I'd like to try testing to confirm it). So, TH9 would be 50%+50% + 7*10% = +170% (and TH10 would be +180%), or 2.7x the baseline drop rate. If the original drop rate was 10%, that means you'd have a 27% drop rate with TH9 and 28% drop rate with TH10.

To distinguish between TH9 and TH10 in that model you'd need about 25,000 samples to reach a margin of error small enough to be reasonably certain of your results, though a few thousand samples at each of multiple points to create a trend line would probably be more useful.

In such a hypothetical model, the 50% tiers would be:
TH0 - baseline
TH1 - +50%
TH2 - +50%
TH7 - +50%
TH12 - +50%

A main thf and a single TH proc (TH7 total) would be the equivalent of an additional 50% TH tier, which is decent, though not exceptional. Since you're already 100% over baseline with TH2, it would be a 25% boost in the number of drops received over what you could get from a /thf. That would not be terribly out of line with the general feeling of the gain from bringing a thf main compared to a /thf mule.

The data for this model was taken from before the level cap raise, though, so there's the potential that the third TH trait thf natively gets could be another +50%, with each TH+1 from gear or procs being +10%. That would make TH7 +190% instead of +150%, and would give 45% more drops than a /thf.

[Lokithor]

TH10 is so ordinary. Real thieves will turn the TH up to 11.

[Agetos]

TH10 is so ordinary. Real thieves will turn the TH up to 11.

truly rofl at that link

[Granny]

so where are the results of the test you are trying to convince us of... This is nuts, gonna tell us they solved a myth but show no results yet go to the trouble of posting...

To top it all off everyone would be a million times more interested in testing between TH2 and TH3+ in the first place, AND actually SEEING the RESULTS, HINT HINT, lols.

[Vold]

What we're dying to know is just how much TH10 wins out over TH2. I just don't even know what the JP are doing to be fussing over TH9 vs 10. Of all things TH, it's TH9 vs TH10.

All this testing and fooling around is all fun and games but at the end of the day it will never provide cold hard 100% facts no matter how high that kill count gets because numbers will always be skewed by some factor that can't be avoided without 24/7 testing by the same person. Therefore I will ask the devs the following question if reps will be so kind as to pass it along:

Who is your daddy, and what does he do?

Can you explain to us why we should shoot for max TH effectiveness without using the response "because the more/higher you have the better" ?

It won't kill anyone for us to know the maths behind it. Some might pout over possibly weak increases per tier but whatev. I think it's pretty safe to say that the player base is always going to do everything they can to up drop chances in the standard LS alliance gig and that means bringing THFs, period. It'd just be darn handy to know for the solo/duo/low man teams out there that are less than serious business. If I can sacrifice my THF to take my war/thf instead for sake of kill speed and know for a fact that it's not totally crippling my drop rates that would be the best thing since pizza on a stick. Not that I'd do it often but if the situation was dire enough I could be convinced to go with the ancient TH2 trait.

I don't know. Shrug. It's been about a decade. I'm ready for some of those "nono" questions to finally be answered after all these years. See what's right and wrong. Laugh it up about fail theories over the years. Etc. Granted, TH procs are fairly new but TH is TH.

[Arcon]

The data for this model was taken from before the level cap raise, though, so there's the potential that the third TH trait thf natively gets could be another +50%, with each TH+1 from gear or procs being +10%. That would make TH7 +190% instead of +150%, and would give 45% more drops than a /thf.

Not sure if you've been following this thread (and the other one), but we already had a fairly large sample for TH0 and TH2 from before the update, which seem to be indicating something else: re-rolls on each tier. And recent testing, while smaller in scale, seems to coincide with that model for TH1 and TH2 only... after that it surprisingly changes style and goes on differently.

However, I believe there's not any kind of static increase (percentual or otherwise), because that would mean they would either have to cap the effect of TH, or the level of TH or the overall drop rate, none of which I believe is true, otherwise you could reach 100% easily on some drops, which I believe isn't possible with any amount of TH, for several reasons.

[Byrth]

In order to see an effect in 500 kills that's significant at the 95% confidence level, the change would have needed to be about 3~5%.