Stockfish: Discuss an upstream architecture decision
This upstream patch https://github.com/official-stockfish/Stockfish/commit/0371a8f8c4a043cb3e7d08b5b8e7d08d49f28324 seems to be a slowdown (3-6%) on several platforms. It should affect both multi variant fishtest (time wasted) and the play strength on lichess.
ppigazzini
All 14 comments
It's every platform just about.
Sent from my iPhone
On Jul 13, 2017, at 6:56 PM, ppigazzini notifications@github.com wrote:
This upstream patch official-stockfish/Stockfish@0371a8f seems to be a slowdown (3-6%) on several platforms. It should affect either multi variant fishtest (time wasted) and the play on lichess.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
MichaelB7
on 14 Jul 2017
I agree with the maintainer's comment. Also, supposing there is any slowdown, that might actually decrease resources spent by fishtest as mistakes will be more common so fewer games will be played.
Is there an issue specific to this repository? Play on lichess uses movetime to constrain time spent per move.
ddugovic
on 14 Jul 2017
Whatever the case may be, you got your wish: d31f068312b88ca9a5a5d188898144cc80793635 . Still dunno whether there was anything specific to this repository that required us to subvert architectural goals of the upstream maintainers.
To clarify, I've raised my concern about the revert in the Stockfish developer forums as I'm shocked the upstream issue is closed in a way that makes Stockfish brittle.
ddugovic
on 14 Jul 2017
Also, supposing there is any slowdown, that might actually decrease resources spent by fishtest as mistakes will be more common so fewer games will be played.
Well, a decrease in speed of n% gives the same result as decreasing time control by n%, so the latter should be preferred, since it saves time and energy. On fishtest the time control is scaled using bench results to ensure same effective TC across different machines.
Sorry, but the title change is inapproriate, IMO. @ppigazzini makes a valid point, especially since it is him who contributes quite a bit of the resources that would potentially be wasted due to the slowdown. Furthermore, he is a valuable contributor to this project, especially regarding the wikis, so no reason to react that way even if his point would be invalid.
ianfab
on 14 Jul 2017
Sorry, and you are correct that @ppigazzini raises a valid concern (surely d31f068312b88ca9a5a5d188898144cc80793635 -- now merged at d31f068312b88ca9a5a5d188898144cc80793635 -- proves the validity of the concern). I may have overreacted since I don't know how things can end well if I become at-odds with upstream maintainers regarding architectural decisions. For example, earlier I didn't revert 2d96e8fbe9b683ff320a197715155381139c7664 although it loses Elo on most platforms (in the interest of correctness on all platforms).
I am still stunned that official-stockfish/Stockfish#1164 was both "fixed" and then reverted so quickly and remains closed! An ideal solution would be to find an Elo-neutral (no slowdown) change which is also future-proof without significantly complicating Stockfish. More realistically, some upstream contributor identifies how GCC 7.1 has a bug/issue specific to Stockfish's PGO build.
Indeed, a n% speed decrease could be tested more efficiently on fishtest by simply reducing the time control (rather than changing the compiler flags).
ddugovic
on 14 Jul 2017
It is more efficient to test functionally identical versions by looking at the run-times for a task.
Time differences can be measured with high accuracy. Speed tests can be done locally. BUT: you then will have to trust the individual tests (testers, set-ups, methods). Fishtest in general also uses local tests but collects, filters and assembles them.
It might be interesting to extend fishtest for run-time examinations (benchmark tests), for various reasons.
alwey
on 14 Jul 2017
It might be interesting to extend fishtest for run-time examinations (benchmark tests), for various reasons.
I have that on my todo list for fishtest for quite some time already, but I have not found the time yet to implement it. It would definitely be a great feature, perhaps also for the main fishtest instance.
ianfab
on 14 Jul 2017
I have that on my todo list for fishtest for quite some time already, but I have not found the time yet to implement it. It would definitely be a great feature, perhaps also for the main fishtest instance.
Great - no hurry ... And actually there are various reasons why I believe it could be done _here_ first.
alwey
on 14 Jul 2017
I raised my concern in upstream github repo (https://github.com/official-stockfish/Stockfish/pull/1165), upstream forum, upstream wiki.
No answers from the maintainer, so I opened this issue to discuss the question for this project and for lichess.
No problem for the changes of the title, I synthesized the long name of the upstream patch choosing the word "questionable", pun intended :)
ppigazzini
on 14 Jul 2017
Ha... OK, I had missed that a few days elapsed without a response in either the upstream issues or the forum. I really wasn't looking to "pick a fight" w/them but I now see something had to be done to get someone's attention.
FWIW they've still not responded to my SPSA speedup either... I suppose they don't mind wasting draws instead of scoring them:
https://groups.google.com/forum/?fromgroups=#!topic/fishcooking/iyAFOdVx4RE%5B1-25%5D
ddugovic
on 14 Jul 2017
@ddugovic: Indeed, they did not respond to your suggestion. I have been working with SPSA some years now. From Joona's perl tuner I derived a version that does not play entire games and is more fine-grained. You can specify the number of moves that will contribute to the result. It generates a weighted result from the changes in score (or rather differences in scores) of the two sides. So it is not 1-0, 0-1 and 1/2-1/2, but rather some value between -2.0 and + 2.0. The factor 2 is used to "normalise" the outcome to roughly match the discrete -1, 0 , 1 range. It plays game pairs, the second game with reversed colors. So this tuner would score draws or insignificantly changes in score differences as 0.0 outcome. Maximum result for a game pair is 4.0. The weight of the score is calculated from a tanh-function. This tuner can be used to examine specific game phases: opening, midgame and endgame. This can be influenced by the choice of opening positions and the number of plies to play.
I think this has much in common with your tuner.
alwey
on 14 Jul 2017
Ah, that's clever. My tuner has a quite simple and easy to understand change... declare that the cost function for SPSA is Elo, and therefore replace:
return $result;
with:
# Calculate Elo loss among (-400, -191, 0, 191, 400) normalized to [-2.0, 2.0]
return (-2.0, -0.955, 0, 0.955, 2.0)[$result + 2];
since Elo declares that a draw is a meaningful result.
ddugovic
on 15 Jul 2017
since Elo declares that a draw is a meaningful result.
Yes, so 2-0 and 1-1 is not equal to twice 1.5-0.5 /wrt strength estimation and algorithm progress. Elo-difference also is a tanh function (or rather inverse thereof).
alwey
on 15 Jul 2017
Well, I just submitted a PR. I'm not optimistic they'll accept it, but we'll see...
ddugovic
on 15 Jul 2017
Related issues
ianfab
·
6Comments
Vinvin20
·
14Comments
ianfab
·
9Comments
Vinvin20
·
12Comments
Truthdoc
·
12Comments
Most helpful comment
Well, a decrease in speed of n% gives the same result as decreasing time control by n%, so the latter should be preferred, since it saves time and energy. On fishtest the time control is scaled using bench results to ensure same effective TC across different machines.
Sorry, but the title change is inapproriate, IMO. @ppigazzini makes a valid point, especially since it is him who contributes quite a bit of the resources that would potentially be wasted due to the slowdown. Furthermore, he is a valuable contributor to this project, especially regarding the wikis, so no reason to react that way even if his point would be invalid.