Treasure Hunter Testing

[Babygyrl]

you know that is pretty cool that you guys actually tested that!! def you should do more tests on other possible "myths" out there lol

[Urteil]

Our development team just figured out that 10 > 9.


It is only uphill from here folks, there is nothing they can't conquer.

No curelock they can't fix.

No menu's they can't make unterribad.

[FrankReynolds]

I don't disagree, I'm just saying people complaining that there's no difference at all is what makes people who could test useful things waste a lot of time just to tell us that "everything is as it should be".

Why do they have to test it at all? They could just ask the people who wrote the friggin code. If the gain is that bad that it requires a 1500+ kill sample to be sure it even exists, then it is a crap ability and they should be telling the Development team to get their crap together. Treasure hunter is the excuse behind every short coming SE puts on Thief. Everybody knows it does "something". The problem is that the "something " that it does is tantamount to pissing on a forest fire.

[Ophannus]

Test what "Enhances Soul Jump" effect does on Lancer Cuissots +2. Nobody has any idea what these do at all.

[Sasaraixx]

He said it would take a considerable amount of time so TH9>10 is probably less than 1% making it essentially useless.

You're still not understanding. He said it would take a considerable amount of time to test TH 9 vs10 by raising it MANUALLY. That is why they used equip that gave TH+9 and TH+10.

[Motenten]

For a sample size of 500 mobs, and assuming the drop rate of pugil scales is somewhere in the neighborhood of 20% (Wiki lists 16% for Sicklemoon Jagils in the same zone, though with a pitiful sample size; not really relevant, just need to know roughly where on the continuum the drop rate is likely to be), the 95% margin of error for that test is 3.5%.

That means that if the drop rate of the two runs differed by less than 3.5% then the test proves, literally, nothing. It is not statistically significant. If the drop rates differed by more than 3.5% but less than 7% there's still a chance that the two populations have identical 'real' rates, and the results are less than trustworthy. If the drop rates differed by more than 7% I'll be extremely surprised.

That said...

2. TH9 vs. TH10
Since it would take a ridiculous amount of time to test this by manually increasing the Treasure Hunter effect to 9 or 10 via attacks and what not

Which basically means you guys are admitting that putting more than TH3 on a mob is useless, which basically means TH+ gear is useless. Give me a reason to bring a thf. (at least you admit TH9+ is useless, so why even implement it, make it cap at TH10).

Camate's statement does not have anything to do with your interpreted meaning. He's saying that if you want to sample 500 kills with TH 9 and TH 10 the old fashioned way, you have to get enough procs to reach those two tiers under normal fighting conditions. That means you'll have excess samples of TH 6/7/8, and maybe 11/12, and have to fight in such a way that the battles will last long enough to get said procs.

Since they were only interested in testing TH9 vs TH10, that's a huge amount of wasted time and effort. Of course if the players want to test those values they have to expend that effort, but the devs can skip past all that tediousness.

None of which has anything whatsoever to do with the usefulness or uselessness of any TH tier.

[FrankReynolds]

You're still not understanding. He said it would take a considerable amount of time to test TH 9 vs10 by raising it MANUALLY. That is why they used equip that gave TH+9 and TH+10.

we learn from this:
A) treasure hunter from gear is the same as treasure hunter from procs.
B) treasure hunter from gear / procs. is so weak that it takes 1000 + kills to see a measurable increase from it.

Great job testing guys, now make them fix it.

Edit: I guess if you plan on killing the same NM 1000+ times, then you will see the benefit of bringing a Thief. That means that the reason Treasure Hunter is considered SOOOOOO powerful is: we will be Killing the same Notorious Monster 1,000,000,000 Times. Thanks for that glimpse into the future of FFXI.

[FrankReynolds]

You're still not understanding. He said it would take a considerable amount of time to test TH 9 vs10 by raising it MANUALLY. That is why they used equip that gave TH+9 and TH+10.

IDC how he did the test. The fact that a test that size was required is the problem. It means that the effect is incredibly small.

Let me give you an example: I'm trying to get an Adamantoise Egg. If a Aspid pop costs $6,000,000, and I need to kill 1,000 of them to see a raise in drops from the trait, how many years will I have to spend killing aspid to see the value in bringing a Thief?

[Motenten]

And you seem to fail to understand the meaning of statistical sample sizes.

For example, suppose that TH9 gave a 20% drop rate on an item, and TH10 gave a 25% drop rate (pretty substantial for only an extra +1 proc when you're already starting at 9). Would you consider that a significant increase in drop rate, implying a significant value in going from TH9 to TH10? I think most people would.

However, 500 kills for each TH level is not enough to statistically separate those two values. 1000 kills is not enough. You'd need about 4000 kills to statistically distinguish a full 5% increase in drop rate (a 25% increase in drops received), and even then all you know is that TH10 > TH9; you still couldn't really be certain of how much of a difference there is.

Your complaint that you need thousands of kills to be sure of a difference between the two TH levels is merely a reflection of someone who doesn't understand statistics. You will *always* need thousands of samples to get any kind of decent understanding of the underlying rates of anything in a game like this. You can only get away from it if you're dealing with data that you know is discreet and precisely repeatable, such as exact haste or enmity values.



Now, to take things further aside:

I've read one bit of testing which implied that TH1 was a 50% increase in drop rate over baseline, TH2 was an additional 50%, and each additional point of TH above that was +10% (a fairly reasonable model, really; I'd like to try testing to confirm it). So, TH9 would be 50%+50% + 7*10% = +170% (and TH10 would be +180%), or 2.7x the baseline drop rate. If the original drop rate was 10%, that means you'd have a 27% drop rate with TH9 and 28% drop rate with TH10.

To distinguish between TH9 and TH10 in that model you'd need about 25,000 samples to reach a margin of error small enough to be reasonably certain of your results, though a few thousand samples at each of multiple points to create a trend line would probably be more useful.

In such a hypothetical model, the 50% tiers would be:
TH0 - baseline
TH1 - +50%
TH2 - +50%
TH7 - +50%
TH12 - +50%

A main thf and a single TH proc (TH7 total) would be the equivalent of an additional 50% TH tier, which is decent, though not exceptional. Since you're already 100% over baseline with TH2, it would be a 25% boost in the number of drops received over what you could get from a /thf. That would not be terribly out of line with the general feeling of the gain from bringing a thf main compared to a /thf mule.

The data for this model was taken from before the level cap raise, though, so there's the potential that the third TH trait thf natively gets could be another +50%, with each TH+1 from gear or procs being +10%. That would make TH7 +190% instead of +150%, and would give 45% more drops than a /thf.

[Lokithor]

TH10 is so ordinary. Real thieves will turn the TH up to 11.