Stockfish: Experiment with ideas from jhellis3/mate_finder
https://fr.lichess.org/VZHy5KHR#15, Nf7? is mate in 9 after Qxf7+.
That's a case of too much selectivity from SF (or lack of extension when in danger).
All moves by white is played with check, it's strange that SF doesn't see the mate.
May be rise the check extension depth ?
Vinvin20
All 59 comments
Checks are only extended if the SEE score is greater or equal to zero. To completely remove this condition might be too much, but I think it could make sense to extend all checks that can not be blocked (knight checks and contact checks) and hence are more forcing. I'll do some tests.
ianfab
on 1 Nov 2016
Yes, good idea !
Vinvin20
on 1 Nov 2016
@ianfab How do you test stockfish strength? Does stockfish have capability to import top games to tune analysis? I'm interested in helping out.
isaacl
on 1 Nov 2016
@isaacl: I use a python script that uses (an outdated version of) python-chess to communicate with Stockfish and test patches in self play. Move legality is not tested for, so I can let it play all variants assuming that Stockfish's move generation is working properly. It might not be optimal, but it is working well so far considering the strength improvements in most variants.
For tuning I use a modified version of Stockfish's SPSA tuner. It does a great job, since all piece values, i.e. also Crazyhouse piece values, have been tuned starting from values that were far from the optimum (in most cases the piece values of standard chess).
ianfab
on 1 Nov 2016
very cool, thanks
isaacl
on 1 Nov 2016
@Vinvin20 Both versions (entend all or only unblockable checks) did not pass.
ianfab
on 2 Nov 2016
Probably not enough depth in other lines :-/
Please publish the final results (score in %).
Vinvin20
on 2 Nov 2016
Extend checks even if SEE score is negative:
LLR: -2.99 (-2.94,2.94) [0.00,20.00]
Total: 1490 W: 709 L: 716 D: 65
Only extend checks with negative SEE if it is a knight or contact check:
LLR: -2.98 (-2.94,2.94) [0.00,20.00]
Total: 1332 W: 626 L: 637 D: 69
No regressions, but no improvements either.
ianfab
on 2 Nov 2016
[0,20] is enormous, that can lead to false conclusions.
Ratings point of view :
1) 400/1490x(709-716) = -1.9 points
2) 400/1332x(626-637) = -3.3 points
So few points to not lose against opponents 300 Elo below is OK for me. Because it plays more solid chess.
Vinvin20
on 2 Nov 2016
I do not think it is a good idea to apply changes to the search without any evidence that it is an improvement (or at least a simplification). Furthermore, it might well be that this oversight was caused by the hash collision issue (that has already been fixed), since I cannot reproduce it with the most recent version.
ianfab
on 2 Nov 2016
OK, good news !
Vinvin20
on 2 Nov 2016
With the new version, another forced mate with unblockable checks (hopefully the human didn't see it until the conclusion).
Look at the analyze graph : https://fr.lichess.org/a4XGGUAO#76
Vinvin20
on 3 Nov 2016
One more forced mate with unblockable checks : https://fr.lichess.org/04lW53Qd#70
Vinvin20
on 3 Nov 2016
One more human mate with only unblockable checks : https://fr.lichess.org/9RZpLOfK#61
Vinvin20
on 4 Nov 2016
I think that this simply is a problem arising with the large branching factor and it is not that surprising that it does not find a mate in 10+. However, if you encounter positions where it does not find a mate in 5 or the like, I would be very interested in these.
ianfab
on 4 Nov 2016
I still to think it would be better to extend more lines with unblockable checks.
It's clearly the area where humans score the greater number of victories against SF level 8.
I promise you : a better score against humans, a more stable score through moves and nicer fireworks during games !
Vinvin20
on 5 Nov 2016
In my experience, blockable checks are often better play. So not sure it makes sense to specifically look for unblockable lines.
isaacl
on 5 Nov 2016
Unblockable checks are, for humans, easy to detect, repeatable and lead to a lost game.
That's called a "weak spot" and when a human finds a weak spot, he can exploit it over and over again (and better and better).
Vinvin20
on 6 Nov 2016
Stockfish already has a mechanism for detecting repeatable threats.
ddugovic
on 6 Nov 2016
Killers system tries some moves before others but don't go deeper to find forced mates as humans do.
Vinvin20
on 6 Nov 2016
Stockfish already has a mechanism for going deeper to find forced mates like humans do.
ddugovic
on 6 Nov 2016
One game that shows there's trouble with eval or extensions (the score should be more stable that shown here ) : https://fr.lichess.org/wyIj4qQX/black#97
Vinvin20
on 6 Nov 2016
ddugovic commented a day ago
Stockfish already has a mechanism for going deeper to find forced mates like humans do.
BTW Stockfish underestimates many winning sacrifices in regular chess. That's even more annoying in Crazyhouze because there's even more winning sacrifices !
Vinvin20
on 7 Nov 2016
One more yesterday : https://fr.lichess.org/xwhYnIKo#49
And it gives a wrong analyse, SF suggest that 25... p@g2 is way better but it's mate with the same threats because of unblockable checks (not managed by SF).
Vinvin20
on 11 Nov 2016
IMHO that's an excellent example of the horizon effect and possibly reason to tune search extension - possibly in response to a drop blocking a check.
ddugovic
on 11 Nov 2016
In theory it could be cause to make a smarter transposition table in terms of lower bounds since 25... P@g5 26. Rxg5+ and 25... P@g2 26. Rxg2+ P@g5 27. Rxg5+ are the same position.
ddugovic
on 11 Nov 2016
Same position on the board but not in hand.
But when there's unblockable checks, the hand of the defender has no effect.
Vinvin20
on 12 Nov 2016
Even if the checks can be blocked, transpositions where the defender has worse pieces in hand must be equal or worse.
ddugovic
on 12 Nov 2016
I don't know hashtables working with a "part" of a position.
Extend more checks seems easier. Or reduce more when the position has less checks.
Or may be it's impossible to exceed the current top human level at 0.5 seconds per move ...
Vinvin20
on 13 Nov 2016
transpositions where defender has worse pieces in hand must be equal or worse.
@ddugovic Do you mean fewer of the same piece types? Hard to make characterizations about different pieces in hand otherwise, it's too positionally dependent. Queen in hand is usually better than pawn in hand, for example, but not always.
isaacl
on 14 Nov 2016
Yes, I meant fewer pieces of the same piece types (and now I see that there are some positions where you need your opponent to block using a specific piece so you can capture it then drop it).
ddugovic
on 14 Nov 2016
I just remember a thing about mate threats, there's a fork of Stockfish way smarter in finding mates : http://www.talkchess.com/forum/viewtopic.php?t=61932
If you could apply this changes to the current SF-zh that would probably solve a lot forced checkmates !
https://github.com/jhellis3/Stockfish/tree/mate_finder
Vinvin20
on 14 Nov 2016
That's a separate issue and I'm seriously considering it although it will weaken Stockfish.
ddugovic
on 14 Nov 2016
Mmmmm...
I'm not sure. SF Matefinder is usually about 40-50 ELO weaker than regular SF.
But it has been optimized for finding tactical shots and long mates.
I'm going to examine the diff from Matefinder SF to regular SF and try to extract the ideas it uses.
I'm guessing that it's going to be worth maybe 100 ELO at crazyhouse.
stockfishdeveloper
on 14 Nov 2016
In my experience Stockfish heavily relies on aggressive pruning in crazyhouse, so I am not sure whether finding some mates a little bit earlier compensates for the increase of the branching factor. But I am more than happy if someone proves me wrong and finds an improvement with these ideas.
ianfab
on 14 Nov 2016
I'm guessing that it's going to be worth maybe 100 ELO at Crazyhouse.
I hope so. It's a complete different world : in Crazyhouse, top amateurs are currently 100 to 200 points (not enough results with long time control) over SF-zh and in regular chess, SF is 300 to 400 over top _professionals_ .
Vinvin20
on 14 Nov 2016
I'm going to examine the diff from Matefinder SF to regular SF and try to extract the ideas it uses.
Please keep me in touch when you have some results against computers and some analyses of tactical games. Results against strong humans would be the most useful but it's not simple to get them.
Vinvin20
on 16 Nov 2016
Ok.
I'm working on it.
stockfishdeveloper
on 16 Nov 2016
Another simple mate in 4 overlooked : 16.P@e7?? Nh3!! :-(
https://fr.lichess.org/X9YlDNv2/black#31
Vinvin20
on 29 Nov 2016
I cannot duplicate that blunder from a command line, and certainly during postmortem analysis Stockfish has no trouble finding it:
setoption name UCI_Variant value crazyhouse
position fen 3r1rk1/ppp4p/3p1n2/7b/4P3/8/PP3PPP/RNB2RK1[QBBPPnnqppp] w - - 30 16
go movetime 800 depth 21
The local engine keep p@e7? until depth=9 and stay a long time in depth=9.
Vinvin20
on 29 Nov 2016
"Local Stockfish" is coded in JavaScript...
This depth=9 test instantly rejects P@e7:
setoption name UCI_Variant value crazyhouse
position fen 3r1rk1/ppp4p/3p1n2/7b/4P3/8/PP3PPP/RNB2RK1[QBBPPnnqppp] w - - 30 16
go movetime 800 depth 9
"Local Stockfish" is coded in JavaScript...
I suppose there's the same logic (search + eval) in the code.
Vinvin20
on 29 Nov 2016
In Chrome, the JS engine plays B@c4 at depth 8
and in Firefox, the JS engine plays P@e7 at depth 8
There's a bug somewhere !
Vinvin20
on 29 Nov 2016
Indeed, I suppose someone ought to enter an issue into Mozilla's tracker...
In all seriousness, "Local Stockfish" isn't Stockfish and I have no control over it.
ddugovic
on 29 Nov 2016
Who does "compile" to JavaScript ?
Vinvin20
on 30 Nov 2016
@Vinvin20 https://github.com/niklasf/stockfish.js
ianfab
on 30 Nov 2016
cc @niklasf
"Local stockfish" should have similar behavior to trunk. On chrome it's a pnacl build, emscripten version is for other browsers.
The likely discrepancies:
- lichess might be sending bad commands, for example with multipv, or misinterpreting results.
- lichess version can be a week or two behind ddugovic trunk.
It's possible to set breakpoints in chrome to monitor communication between stockfish and the browser. That's most accurate info for debugging a miseval.
fwiw I don't see the reported issue, on my chrome I get N@h3.
isaacl
on 1 Dec 2016
The latest version (2016-11-30) seems OK under Firefox now.
Vinvin20
on 1 Dec 2016
Here is a difficult test position (mate in 30 plies):
setoption name UCI_Variant value crazyhouse
setoption name Threads value 4
position fen r5k1/pppqbrp1/2n3Bp/3p1n1p/4p3/1PN1P2B/P1PP2PP/R1B2RK1[NPq] b - - 33 17
go infinite
When you add "moves Q@h1" it finds mate in 15 more than 100 times faster (no exaggeration). Of course, a queen sac will not be considered immediately, but the difference by adding this halfmove should not be that large.
ianfab
on 2 Dec 2016
You are correct (searchmoves Q@h1 with Threads=4):
info depth 14 seldepth 29 multipv 1 score mate 16 nodes 1001969 nps 2745120 tbhits 0 time 365 pv Q@h1 g1h1 f5g3 h2g3 d7h3 g2h3 f7f1 N@g1 f1g1 h1g1 N@f3 g1f1 R@h1 f1g2 h1g1 g2f2 B@e1 f2e2 f3d4 e3d4 c6d4 e2e3 d4c2 e3e2
...
info depth 17 seldepth 29 multipv 1 score mate 15 nodes 1061056 nps 2755989 tbhits 0 time 385 pv Q@h1 g1h1 f5g3 h2g3 d7h3 g2h3 f7f1 N@g1 f1g1 h1g1 N@f3 g1f1 R@h1 f1g2 h1g1 g2f2 B@e1 f2e2 f3d4 e3d4 c6d4 e2e3 d4c2 e3e2 P@d3 e2d1 c2e3 d2e3
I'm now trying adding searchmoves e7d6 Q@h1 to see to what degree results change (Threads=4)...
info depth 22 seldepth 35 multipv 1 score cp 189 lowerbound nodes 373486043 nps 2778087 hashfull 999 tbhits 0 time 134440 pv Q@h1
...
info depth 22 seldepth 35 multipv 1 score mate 19 lowerbound nodes 429695835 nps 2776440 hashfull 999 tbhits 0 time 154765 pv Q@h1
...
info depth 22 seldepth 35 multipv 1 score mate 15 nodes 431691322 nps 2776864 hashfull 999 tbhits 0 time 155460 pv Q@h1 g1h1 f5g3 h2g3 f7f1 N@g1 d7h3 g2h3 f1g1 h1g1 N@f3 g1f1 R@h1 f1g2 h1g1 g2f2 B@e1 f2e2 c6d4 e3d4 f3d4 e2e3 d4c2 e3e2 P@d3 e2d1 c2e3 d2e3 P@c2
and now matefinder with setoption name Study value true adding searchmoves e7d6 Q@h1 to see to what degree results change (Threads=4)...
info depth 21 seldepth 32 multipv 1 score cp 135 nodes 1183644465 nps 2711168 hashfull 1000 tbhits 0 time 436581 pv e7d6 P@f4 h5h4 N@e2 c6e7 g6f7 g8f7 R@e5 B@e6 e5e6 d7e6 e2d4 f5d4 h3e6 d4e6 Q@h5 B@g6 h5h4 R@h5 h4g4 N@f6 g4e2 Q@h4 g2g3 h4h2 e2h2
...
info depth 22 seldepth 32 multipv 1 score cp 558 lowerbound nodes 1259743098 nps 2712058 hashfull 1000 tbhits 0 time 464497 pv Q@h1
...
info depth 22 seldepth 32 multipv 1 score mate 21 lowerbound nodes 1844066824 nps 2719593 hashfull 1000 tbhits 0 time 678067 pv Q@h1
...
info depth 22 seldepth 36 multipv 1 score mate 15 nodes 1853484486 nps 2720375 hashfull 1000 tbhits 0 time 681334 pv Q@h1 g1h1 f5g3 h2g3 f7f1 N@g1 d7h3 g2h3 f1g1 h1g1 N@f3 g1f1 R@h1 f1f2 B@g1 f2e2 f3d4 e3d4 c6d4 e2e1 g1e3 N@g1 h1g1 N@f1 P@f2 e1d1 g1f1 R@e1 f2e1q
One more mate, posted by mathace in the forum today : https://fr.lichess.org/forum/game-analysis/crazyhouse-puzzle#1
Position : https://fr.lichess.org/FbpWrfbW/black#33
How much time to find Q@h1!! ?
Vinvin20
on 2 Dec 2016
(Closed by misclick)
Vinvin20
on 2 Dec 2016
I linked to that crazyhouse puzzle an hour ago.
ddugovic
on 2 Dec 2016
Sorry, I didn't see.
But it's totally related to my tactical test suite where Stockfish-chess overlooks a lot of winning sacrifices.
Vinvin20
on 2 Dec 2016
Are there any other positions in that test suite?
ddugovic
on 2 Dec 2016
There's 161 positions but only for regular chess.
Vinvin20
on 2 Dec 2016
Since the mate_finder branch is created and is much slower than master, we can continue the discussion on #148 .
ddugovic
on 3 Dec 2016
Related issues
ddugovic
路
10Comments
ddugovic
路
9Comments
Nordlandia
路
13Comments
ppigazzini
路
14Comments
Vinvin20
路
11Comments
Most helpful comment
I do not think it is a good idea to apply changes to the search without any evidence that it is an improvement (or at least a simplification). Furthermore, it might well be that this oversight was caused by the hash collision issue (that has already been fixed), since I cannot reproduce it with the most recent version.