Stockfish: Looking for improvements in Crazyhouse

[Vinvin20]

Crazyhouse is surely the most complex variant (for computers at least) on Lichess. I gathered some losts by SF against humans in the last days, that could bring some enlightenment to improve the engine :

3 wins by mastertan
https://fr.lichess.org/CGY1aSdp/black
https://fr.lichess.org/6gTspqQP
https://fr.lichess.org/9RZpLOfK

1 win by chickencrossroad : https://fr.lichess.org/rlVBIWq8/black#0

A lot of wins by opperwezen (links with white and black) :

White : https://fr.lichess.org/games/search?players.a=opperwezen&players.b=&players.white=opperwezen&players.black=&players.winner=&ratingMin=&ratingMax=&hasAi=1&aiLevelMin=8&aiLevelMax=&source=&perf=18&mode=&turnsMin=&turnsMax=&durationMin=&durationMax=&clock.initMin=&clock.initMax=&clock.incMin=&clock.incMax=&status=&winnerColor=1&dateMin=4d&dateMax=&sort.field=d&sort.order=desc

Black : https://fr.lichess.org/games/search?players.a=opperwezen&players.b=&players.white=&players.black=opperwezen&players.winner=&ratingMin=&ratingMax=&hasAi=1&aiLevelMin=8&aiLevelMax=&source=&perf=18&mode=&turnsMin=&turnsMax=&durationMin=&durationMax=&clock.initMin=&clock.initMax=&clock.incMin=&clock.incMax=&status=&winnerColor=2&dateMin=4d&dateMax=&sort.field=d&sort.order=desc

[Yery]

Very quick analysis:

First game: missed danger of queen sacrifice (so king safety too low, or overestimation value of queen?)
Second game: too many 'holes' and ruined pawn structure. Also allowed queen sacrifice which now created one more hole in blacks defense. Finally starts kingwalking at move 23?? why not capture with knight?
Third game: no obvious blunder, but black should develop its pieces (especially knights) early in the game, after a few moves it is simply too late, and these pieces remain death for the rest of the game. For instance on move 6, it could have developed its N.
Forth game: again opening mistake. It seems to like developping its N on the 2/7 rank thereby blocking bishop and after that bishop remains blocked for most of the game. Also later on, it prefers to kingwalk rather than putting pieces on the board for defense. It is good to have a few pieces in hand, but when in danger, it is good to put them on the board.
Fifth game, first opperwezen: again development. As a general use: get the d and e pawns moving early on, and get your bishops out. Then there is incomprehensible https://fr.lichess.org/fJvoPWs9#33 ???
Sixth game: again development, again bishop closed in,
Seventh game: same: king's bishop and knight left undeveloped.
Eight game: ends up with 2 queens against a LOT of pieces, and clearly overevaluates own position because of it, queen overevaluated!
Ninth game: similar, so 2 queens, and also 2 undeveloped pieces on the king side

[Vinvin20]

Thanks for this nice insight, Yery !

[Vinvin20]

7 more losts by Level 8 in last days :
https://fr.lichess.org/uhwK8rbO
https://fr.lichess.org/8CKF5BAL#49
https://fr.lichess.org/OMHyaPqi
https://fr.lichess.org/AVyk3kVR/black
https://fr.lichess.org/xwhYnIKo
https://fr.lichess.org/VkMjMa4A
https://fr.lichess.org/u8Zm4MBN

[Yery]

Does lichess update the engine regulary with the new commits? These are in my opinion the two largest repeated errors:

• opponent manages to get a reasonable safe position with a huge number of pieces in hand: game 1, 2 and 3, 5 and 6 (so SF should defend more in such cases and seems to misevaluate its own kingsafet), - question is why does it let it come this far?, why keep attacking and exchanging pieces like a madman? If the opponent has a huge reservoir of pieces, own safety becomes very important.
• Undeveloped pieces: Game 3 N+B , Game 4 N, Game 5 N+2xB!!!!, Game 7 (here too Q out too early)

Thanks a lot for these links: keep them coming!

[ianfab]

@Yery Changes are usually merged only every few weeks to ensure stability, so a few improvements, e.g. this one dealing with king safety, are not yet on lichess.

Since the mobility of pieces is more important in crazyhouse than in standard chess and since we so far use the same mobility bonuses as for standard chess, it underestimates the importance of developing pieces. Simply doubling the mobility bonus so far performs quite well in a test, so I might commit this tomorrow if it passes the full test. This simple "fix" will eventually be replaced by tuned values, but tuning always takes a lot of computation time.

Thanks for analyzing the games. Condensing the observations to a few simple ideas like king safety or mobility helps a lot.

[Vinvin20]

May be add a mobility bonus for pieces in hand because they can go on a lot of squares !

[ianfab]

@Vinvin20 In the commit linked above this is accounted for by adding a bonus for pieces in hand to compensate for the piece-square table and mobility bonuses.

[Vinvin20]

This simple "fix" will eventually be replaced by tuned values, but tuning always takes a lot of computation time.

I could help you in running tuning sessions. My computer ( 6*4 GHz ) is not use often.
I use windows and I never ran tuning session before, so you'll to explain me clearly how to proceed.

[ianfab]

@Vinvin20 Thanks, that would be nice. First, you could try to get local tuning of official Stockfish to work, and from there it only requires minor changes.

1. Go through the steps at https://github.com/glinscott/fishtest/wiki/Creating-my-first-test#tuning-with-spsa up to step 10 (the rest is not necessary for local tuning) to obtain a tunable executable of Stockfish.
2. Download the SPSA tuner and start a tuning session as explained in the readme using the executable from step 1.

If this works you can use this Stockfish fork instead of the official repository in step 1 (but merge the tuning branch from the official repository) and use my fork of the SPSA tuner.

[Vinvin20]

OK, I'll look at that !

[Vinvin20]

Turmoil today ! JannLee, the very top zh player, challenged SF lvl8 with comments ! And won 8-2.
https://www.youtube.com/watch?v=VTNAHmlQDXA

[ianfab]

@Vinvin20 Thanks for posting the link. Although it is the version from more than a week ago, it will still be interesting to listen to his comments.

[Yery]

I had a look at the first two games, and I observe the same mistakes as pointed out above - I hope lichess updates their version soon!

In addition, it seems that it doesn't sufficiently use its time (often it just moves immediately, see e.g. the first game, where it gets checkmated with more than 6 minutes on its clock). Does it only play on its increment?

[ianfab]

@Yery For each level it has a fixed time limit per move regardless of the human player's time control. See https://en.lichess.org/blog/U4mtoEQAAEEAgZRL/strongest-chess-player-ever.

[Vinvin20]

@Vinvin20 Thanks, that would be nice. First, you could try to get local tuning of official Stockfish to work, and from there it only requires minor changes.

I've not installed anything for github yet, do I have to install this for starting ?
https://github.com/glinscott/fishtest/wiki/A-portable-fishtest-worker-for-Windows-64-bit

[ianfab]

@Vinvin20 No, this is not needed for local tuning/testing. You only need git, Stockfish sources, and the SPSA tuner (which requires perl). I haven't used git on windows so far, but probably this will do.

[Vinvin20]

@Vinvin20 No, this is not needed for local tuning/testing. You only need git, Stockfish sources, and the SPSA tuner (which requires perl). I haven't used git on windows so far, but probably this will do.

I give up trying to make this working, there's not enough explanations.

[Vinvin20]

14 more losts in the last couple of days :

5 by mathace :
https://fr.lichess.org/@/mathace/search?dateMin=&dateMax=&ratingMin=&ratingMax=&turnsMin=&turnsMax=&durationMin=&durationMax=&clock.initMin=&clock.initMax=&clock.incMin=&clock.incMax=&source=&perf=&mode=&hasAi=1&aiLevelMin=8&aiLevelMax=&players.a=mathace&players.b=&players.winner=mathace&players.white=&players.black=&status=&winnerColor=&sort.field=d&sort.order=desc

2 by VarnaGX :
https://fr.lichess.org/@/VarnaGX/search?dateMin=&dateMax=&ratingMin=&ratingMax=&turnsMin=&turnsMax=&durationMin=&durationMax=&clock.initMin=&clock.initMax=&clock.incMin=&clock.incMax=&source=&perf=&mode=&hasAi=1&aiLevelMin=&aiLevelMax=&players.a=varnagx&players.b=&players.winner=varnagx&players.white=&players.black=&status=&winnerColor=&sort.field=d&sort.order=desc

7 by xuanet :
https://fr.lichess.org/@/xuanet/search?dateMin=&dateMax=&ratingMin=&ratingMax=&turnsMin=&turnsMax=&durationMin=&durationMax=&clock.initMin=&clock.initMax=&clock.incMin=&clock.incMax=&source=&perf=&mode=&hasAi=1&aiLevelMin=&aiLevelMax=&players.a=xuanet&players.b=&players.winner=xuanet&players.white=&players.black=&status=&winnerColor=&sort.field=d&sort.order=desc

Please, update the Crazyhouse engine on Lichess !

[Vinvin20]

One more chaotic position to analyze for SF : https://fr.lichess.org/NoCVf58d/black#78

[MichaelB7]

getting very good results with futility_margin_factor of 125 ( better than 150 - tried various 300, 75 125 and 175) 125 was by far the best. Cant submit a PR since I'm already forked to Stockfish - s/b near line line 100 in search.c

  const int futility_margin_factor[VARIANT_NB] = {

#ifdef CRAZYHOUSE
  125

Rank Name                                          Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
1 Stockfish 8-112516-CrazyHouse 64 POPCNT (UCI2WB)   3162   0.0   65   65   100   65.0  65.0   63   33    4  63.0   4.0  3038 
2 Stockfish 251116 64 POPCNT (UCI2WB)                3038 123.2   65   65   100   35.0  35.0   33   63    4  33.0   4.0  3162

[ianfab]

A few weeks ago, I tuned futility_margin_factor and the result was 138, but it did not pass an SPRT test. Your results look promising, but they are from only 100 games, so I will test both, 125 and 138, with more games to get sufficient statistical data.

[ianfab]

Currently, the intermediate result of futility_margin_factor 125 vs. 150 is
Total: 1000 W: 479 L: 466 D: 55.

[ddugovic]

Also be aware that an upstream pull request is coming which I think might affect this... :-(

[ianfab]

I do not think that this considerably affects the futility margin, but I might be wrong.

Anyway, the tests have finished:

futility_margin_factor = 125:
LLR: -3.01 (-2.94,2.94) [0.00,20.00]
Total: 2190 W: 1048 L: 1035 D: 107

futility_margin_factor = 138:
LLR: -3.06 (-2.94,2.94) [0.00,20.00]
Total: 1588 W: 750 L: 755 D: 83

[Vinvin20]

A couple of more wins by Opperwezen and Chickencrossroad :

https://fr.lichess.org/@/opperwezen/search?dateMin=&dateMax=&ratingMin=&ratingMax=&turnsMin=&turnsMax=&durationMin=&durationMax=&clock.initMin=&clock.initMax=&clock.incMin=&clock.incMax=&source=&perf=&mode=&hasAi=1&aiLevelMin=&aiLevelMax=&players.a=opperwezen&players.b=&players.winner=opperwezen&players.white=&players.black=&status=&winnerColor=&sort.field=d&sort.order=desc

https://fr.lichess.org/@/chickencrossroad/search?dateMin=&dateMax=&ratingMin=&ratingMax=&turnsMin=&turnsMax=&durationMin=&durationMax=&clock.initMin=&clock.initMax=&clock.incMin=&clock.incMax=&source=&perf=&mode=&hasAi=1&aiLevelMin=&aiLevelMax=&players.a=chickencrossroad&players.b=&players.winner=chickencrossroad&players.white=&players.black=&status=&winnerColor=&sort.field=d&sort.order=desc

[MichaelB7]

Here' another option to try , I ran these tests twice , one using an opening book (white and black ) and once using no book - almost identical results with and without book:

in search.cpp - make the changes - change 13 to 15 (depth >= 13) and 12 (depth < 12) to 14.

#ifdef HORDE
     if (pos.is_horde()) {} else
 #endif
-    if (   !PvNode
+    if (!PvNode
         &&  eval >= beta
-        && (ss->staticEval >= beta - 35 * (depth / ONE_PLY - 6) || depth >= 13 * ONE_PLY)
+        && (ss->staticEval >= beta - 35 * (depth / ONE_PLY - 6) || depth >= 15 * ONE_PLY)
         &&  pos.non_pawn_material(pos.side_to_move()))
     {
         ss->currentMove = MOVE_NULL;
@@ -8794,7 +8818,7 @@
             if (nullValue >= VALUE_MATE_IN_MAX_PLY)
                 nullValue = beta;

-            if (depth < 12 * ONE_PLY && abs(beta) < VALUE_KNOWN_WIN)
+            if (depth < 14 * ONE_PLY && abs(beta) < VALUE_KNOWN_WIN)
                 return nullValue;

             // Do verification search at high depths

---

Rank   Name                                        Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR 
1      Stockfish 281116 64 POPCNT (UCI2WB)           3114   0.0   17   17  1424  757.5  53.2  747  656   21  52.5   1.5  3086 
2      Stockfish variant-master 64 POPCNT (UCI2WB)   3086  27.2   17   17  1424  666.5  46.8  656  747   21  46.1   1.5  3114 

[ddugovic]

For reference, the concept we're referring to is:

// Step 8. Null move search with verification search (is omitted in PV nodes)

and suggesting to do less of it, noting that in Horde chess we don't bother doing null-move search at all.

I'd need to ensure the different constants are only used for crazyhouse... and yet I wonder whether null-move search is useful in crazyhouse, since in so many positions a "pass" or "do-nothing" move sharply changes the evaluation.

[ianfab]

Since I also wondered about the value of null move search in crazyhouse, I tested a version where it is skipped and the result was:

LLR: -3.02 (-2.94,2.94) [0.00,20.00]
Total: 3980 W: 1938 L: 1874 D: 168

So it is a small Elo gain even though it did not pass the test. However, adequately reducing null move search should perform much better than completely skipping it.

[Vinvin20]

400/3980*( 1938 - 1874 ) = + 6.4 Rating points

Note it's a simplification, on the SF testing facility, bounds are : [-3.00,1.00]

[ianfab]

Yes, but the purpose of the test was not really to commit this, but to find out how much the current implementation of null move search is worth for crazyhouse, since in other variants it can easily amount to plus or minus 200 Elo. I can not use similar bounds as fishtest, since it would take forever to run such tests on my two machines. However, for simplifications I should indeed use different bounds, maybe something like [-10, 5].

By the way, since I have made a lot of changes (bugfixes and improvements) in the last 48 hours, I have run a regression test for crazyhouse of the current version (1cac3a4) against the one from two days ago (1fd4ce8):

ELO: 124.60 +-22.4 (95%) LOS: 100.0%
Total: 1000 W: 650 L: 306 D: 44

[Vinvin20]

ELO: 124.60 +-22.4 (95%) LOS: 100.0%
Total: 1000 W: 650 L: 306 D: 44

wonderful !!

It's time to challenge JannLee in some blitz games (not only 1 second for SF-zh, and not only with 1 core).
When will this version be in use on Lichess local engine ?

[MichaelB7]

That's a pretty nice gain there.

[Vinvin20]

and suggesting to do less of it...
and yet I wonder whether null-move search is useful in crazyhouse, since in so many positions a "pass" or "do-nothing" move sharply changes the evaluation.

May be do null moves only when there's less than 3 pieces in (sum of both) hands. May be add a condition to be sure kings are safe.
Null moves are interesting to find threats in still positions.

[ddugovic]

131 Finals are imminent and our submission is (probably) final and according to rapid rating list is a heavy favorite. That said I'll continue testing #148 changes (which so far do not scale) after other tests.

Combined Vinvin suggestion with #148. Closing this issue.