Stockfish: Strange results for Crazyhouse in regression test

[Vinvin20]

This results shows +25 Elo since v080217 : http://35.161.250.236:6543/tests/view/58a4e6406e23db7ae0be5001

But the improvement "fix move encoding related TT bug" show around +80 Elo improvement.
http://35.161.250.236:6543/tests/view/58a090f76e23db5f9ce6a01d
http://35.161.250.236:6543/tests/view/58a073b86e23db5f9ce6a016

That means that other mods lose 55 Elo ??

[sf-x]

But the improvement "fix move encoding related TT bug" show around +80 Elo improvement.

No, it doesn't. Stop treating SPRT results as if they were results of a test with number of games fixed in advance.

There was a regression due to a blind merge: https://github.com/ddugovic/Stockfish/commit/2ca8f45219cf4f7484ad2546acb264787f481248 ( http://35.161.250.236:6543/tests/view/5890d4a16e23db6a8dac3938 )

[ddugovic]

I entirely agree with sf-x.

Sometimes upstream changes improve Stockfish at variants and sometimes they regress Stockfish at variants; however I suspect in the long term that accepting upstream changes will be beneficial even if sometimes regressions occur.

A different policy (which becomes complicated and computationally expensive) is during each merge to distrust upstream tests and cherry-pick which upstream changes to apply (including from previous unmerged changes -- and might require un-merging some previous changes in order to merge others); but this would miss upstream improvements for standard chess and could require re-tuning.

[Vinvin20]

But the improvement "fix move encoding related TT bug" show around +80 Elo improvement.
No, it doesn't. Stop treating SPRT results as if they were results of a test with number of games fixed in advance.

I know for a long time that SPRT is not a regular match (I was one of the first to point that on some forum).
But on 1000 games, it's clear that the perf of the 2 matches is very close to a regular match, so my statement "But the improvement "fix move encoding related TT bug" show around +80 Elo improvement." is very correct .

[ianfab]

@Vinvin20 In the regression test, the release from February 8 was tested against the one from January 5. The mentioned improvement was commited after the latest release, so this will only show up in the regression test of the next release. So I do not see an issue here.

@ddugovic I think that in cases where we can easily avoid upstream changes to affect variants, e.g. tweaks of evaluation parameters where we already have an array for variants, we should not change the values for variants even if they are the same as for standard chess. In other situations, where code would have to be added to keep the previous functionality for variants, I agree that we should apply the upstream changes and test reversions with normal [0,10] bounds in case we suspect a noticeable regression by the merge. In my opinion, this is the most practical approach.

[Vinvin20]

@Vinvin20 In the regression test, the release from February 8 was tested against the one from January 5. The mentioned improvement was commited after the latest release, so this will only show up in the regression test of the next release. So I do not see an issue here.

OK, so the "around" +80 Elo improvement will come ! Thanks !

[ddugovic]

my statement "But the improvement "fix move encoding related TT bug" show around +80 Elo improvement." is very correct .

OK, so the "around" +80 Elo improvement will come ! Thanks !

Look, I don't want to have to mark this repository as "beta" just to get across the idea that improvements in one release aren't 100% guaranteed to appear in exactly the same magnitude in the next release... also, SPRT tests are not normal matches with a preset number of games. Have you studied statistics at all? If not I encourage it, thanks.

[Vinvin20]

Have you studied statistics at all? If not I encourage it, thanks.

I'll take a simple example : If we would have chose to play 458 games we would have get the result as here :
https://github.com/ianfab/Stockfish/compare/4c8abf5...ed12559

Total: 458 W: 264 L: 162 D: 32

Do you think the results would have been very different if we would have chose to play 400 or 450 or 500 or 600 or 1000 games ?

[ddugovic]

I'll take a simple example...

There is nothing simple about SPRT. Honestly if you had done any study in statistics you would appreciate how easy it is to misinterpret raw data: http://xkcd.com/882/

[stockfishdeveloper]

@Vinvin20 SPRT is a statistical tool that determines confidence intervals for a given dataset.
The way we use SPRT, for determining whether to commit a patch, is only one use for it.
We want to make sure that our changes actually improve Stockfish or at least don't regress.
So what we do is say "I want to be x percent sure that this change will be worth at least y ELO or at worst, lose z ELO". Usually the bounds, y and z, are +10 and -0 for functional variant testing. The "x percent sure" is the confidence interval. Usually this is set at 95. This means that we want to be 95% sure of the result of the patch, whether it gains or loses ELO.

Does this makes sense?

[Vinvin20]

Yes, I already know all of this.

If we would have chose to play 458 games and the games was the same as there, we would have get the same result as the SPRT : +70 Elo.
Some people have troubles to understand this...

[ddugovic]

By definition SPRT plays until an extreme result occurs. If you were to play 458 games 1000 times, 950 of those times a less significant result would occur.

[stockfishdeveloper]

@Vinvin20 "If we would have chose to play 458 games" shows me that you don't know how SPRT works.
We don't know beforehand how many games will be played.
See: https://en.wikipedia.org/wiki/Sequential_probability_ratio_test

[Vinvin20]

...shows me that you don't know how SPRT works.
We don't know beforehand how many games will be played.

Please, don't blame me for something I didn't write.

[ianfab]

As expected, the new regression test shows about +100 Elo. http://35.161.250.236:6543/tests/view/58b038226e23db1e93867044