Leela-chess: Endgame performance evolution

[evalon32]

I thought it'd be fun to start tracking the network's performance on basic checkmates as it evolves. (I'm not trying to train it, just to evaluate the different generations.) Here's what I've got for now:

Network | KQQvK | KQRvK | KRRvK | KQvK | KRvK | KBBvK | KBNvK
--- | --- | --- | --- | --- | --- | --- | ---
kbb1-64x6-796000 | 100 | 100 | 100 | 94 | 69 | 12 | 3
random-8x0 | 40 | 31 | 49 | 10 | 12 | 0 | 0
seed-64x6 | 100 | 92 | 67 | 50 | 12 | 0 | 1
gen2 (de1ddecb) | 96 | 92 | 92 | 33 | 18 | 0 | 0

Methodology: the score is the number of wins out of 100 selfplay games (with noise, no randomize, default no. of playouts). The starting position was generated with Position::set(code, color, si), except that I modified it to place the pieces on the center files. For example, KBNvK uses FEN "8/3k4/8/8/8/8/3KBN2/8 w - - 0 10"

Edit: Added the current network, and also the SL-trained one, for reference. Note that since this is selfplay, a drop in winrate doesn't necessarily mean a regression. gen2 probably learned more about escaping a queen mate than about giving one.

[Error323]

Cool! definitely an interesting metric!

[glinscott]

That's great to see :).

[killerducky]

de1ddecb is very strong at running away from the Queen!

[evalon32]

Now with gen3.

Network | KQQvK | KQRvK | KRRvK | KQvK | KRvK | KBBvK | KBNvK
--- | --- | --- | --- | --- | --- | --- | ---
kbb1-64x6-796000 | 100 | 100 | 100 | 94 | 69 | 12 | 3
random-8x0 | 40 | 31 | 49 | 10 | 12 | 0 | 0
gen1 | 100 | 92 | 67 | 50 | 12 | 0 | 1
gen2 | 96 | 92 | 92 | 33 | 18 | 0 | 0
gen3 | 100 | 95 | 94 | 44 | 19 | 0 | 0

[evalon32]

New stats!
First, gen4 and gen5. I also measured the average number of moves to mate.

Network | KQQvK | KQRvK | KRRvK | KQvK | KRvK | KBBvK | KBNvK
--- | --- | --- | --- | --- | --- | --- | ---
_perfect play_ | _100 / 2_ | _100 / 3_ | _100 / 4_ | _100 / 7_ | _100 / 11_ | _100 / 14_ | _100 / 27_
kbb1 | 100 / 9.1 | 100 / 11.9 | 100 / 13.0 | 94 / 16.9 | 69 / 25.6 | 12 / 29.9 | 3 / 18.0
rng | 40 / 8.6 | 31 / 15.5 | 49 / 13.8 | 10 / 22.2 | 12 / 16.1 | 0 / inf | 0 / inf
gen1 | 100 / 9.2 | 92 / 17.9 | 67 / 26.5 | 50 / 26.0 | 12 / 29.8 | 0 / inf | 1 / 25.0
gen2 | 96 / 22.5 | 92 / 16.1 | 92 / 15.7 | 33 / 36.7 | 18 / 27.7 | 0 / inf | 0 / inf
gen3 | 100 / 13.5 | 95 / 17.9 | 94 / 11.1 | 44 / 25.5 | 19 / 26.5 | 0 / inf | 0 / inf
gen4 | 100 / 7.0 | 94 / 16.0 | 93 / 14.4 | 56 / 29.6 | 29 / 38.1 | 0 / inf | 0 / inf
gen5 | 100 / 6.9 | 98 / 11.9 | 99 / 10.1 | 58 / 26.2 | 28 / 37.6 | 1 / 29.0 | 0 / inf

Second, I hypothesized that gen2 got better at defending in KQvK and that's why the win rate dropped. Evidence shows otherwise :) Here are the results of a KQvK round-robin tournament between the six network generations (20 games / side):

KQ v K | rng(B) | gen1(B) | gen2(B) | gen3(B) | gen4(B) | gen5(B) | Total
--- | --- | --- | --- | --- | --- | --- | ---
rng(W) | | 1 | 2 | 1 | 1 | 0 | 5
gen1(W) | 8 | | 15 | 6 | 12 | 10 | 51
gen2(W) | 2 | 2 | | 1 | 3 | 0 | 8
gen3(W) | 12 | 8 | 7 | | 5 | 6 | 38
gen4(W) | 7 | 9 | 11 | 7 | | 9 | 43
gen5(W) | 12 | 9 | 14 | 6 | 13 | | 54

[Error323]

Nice work! We seem to be making good progress as evident both from elo and these metrics! The averages moves to mate are a nice addition!

@evalon32 are you using the default of 800 playouts?
edit: ah sorry you clearly state the conditions in the top post :)

[evalon32]

@Error323 Yes, everything is with 800 playouts.

[evalon32]

gen6: Big jump in KQvK and KRvK, first to get a perfect score in KQRvK, and also the first to score >0 with KQvK against Stockfish (based on limited testing: 3 out of 10 for gen6, 0 out of 10 for rng through gen5).

EDIT: Added gen7

Btw: is there some way I can tell cutechess _not_ to alternate colors (make Stockfish Black in every game)? I tried -rounds n -games 1, didn't help.

EDIT: Added gen8
EDIT: Added gen9, gen10

Network | KQQvK | KQRvK | KRRvK | KQvK | KRvK | KBBvK | KBNvK
--- | --- | --- | --- | --- | --- | --- | ---
gen6 | 100 / 8.8 | 100 / 16.3 | 98 / 18.6 | 84 / 35.0 | 57 / 36.7 | 1 / 29.0 | 1 / 24.0
gen7 | 100 / 15.5 | 100 / 16.0 | 99 / 18.3 | 98 / 31.8 | 78 / 37.3 | 0 / inf | 0 / inf
gen8 | 100 / 11.0 | 100 / 16.0 | 100 / 14.6 | 97 / 35.1 | 74 / 33.5 | 1 / 38.0 | 0 / inf
gen9 | 99 / 11.4 | 100 / 25.7 | 99 / 24.3 | 100 / 34.1 | 42 / 34.9 | 1 / 38.0 | 1 / 48.0
gen10 | 99 / 10.3 | 92 / 25.4 | 94 / 28.3 | 84 / 31.6 | 29 / 34.2 | 0 / inf | 0 / inf

[glinscott]

Nice graph! Exciting to see it learn the basic endgame techniques :).

[zz4032]

@evalon32
" -noswap Do not swap sides of paired engines"

[evalon32]

Thanks @zz4032, exactly what I needed.

[Uriopass]

What about gen9 and gen10 ? @evalon32

EDIT : Wow, the downfall of the KRvK is impressive.

[evalon32]

@Uriopass Just added them a minute ago :)

[jkiliani]

I think this information for gen9 and gen10 is highly relevant, and indicates to me that the net is forgetting some basic mating patterns. I think a (considerably) larger window size may work best here. From the graph I'd suggest trying all the data that went into gen8 plus everything newer, this should amount to all games played since gen6.

The alternative would be to let the net figure these things out again, but I think this will take longer...

[sf-x]

"never" is a curious typo... is it "ever" or "newer"?

[mooskagh]

With the fixed window of 100000 games, it seems that accumulating of
40000-70000 games between network update may be a bad thing (as it spans
games of too few generations).
The highest jump in rating was after training on an epoch of 12000 sample
games.

Maybe instead of just taking last 100000 games, it makes sense to take
samples from all epochs? (weighted more towards the latest ones, but still
having non-zero chance of including any of the earlier ones)?

On Mon, Mar 19, 2018 at 3:20 PM sf-x [e-mail] wrote:

"never" is a curoius type... is it "ever" or "newer"?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/glinscott/leela-chess/issues/92#issuecomment-374228286,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKvplzmuzcwSR38-UIsmBZ2v-c6EJkBSks5tf76lgaJpZM4Sa_Sj
.

[Error323]

started new training:

dataset:
  input: /mnt/storage/home/fhuizing/chess/distributed-dataset/run1/
  num_chunks: 200000
  train_ratio: 0.9
gpu: 0
model:
  filters: 64
  residual_blocks: 6
name: online-64x6
training:
  batch_size: 2048
  lr_boundaries:
  - 35000
  lr_values:
  - 0.04
  - 0.002
  path: /mnt/storage/home/fhuizing/chess/networks
  policy_loss_weight: 1.0
  shuffle_size: 1048576
  total_steps: 60000
  value_loss_weight: 1.0

We'll know in a few hours.

[sf-x]

started new training:

on what games? those by the latest generation, or including the previous too?

[Error323]

It always takes the last num_chunks amount of games from the pool. This is everything up to and including gen6 basically.

[Error323]

I should note however, that I also set move_count to 0. So this input plane will no longer be holding interesting information.

[JackThomson2]

Are we using momentum during training?

[Error323]

yes, see https://github.com/glinscott/leela-chess/blob/master/training/tf/tfprocess.py#L121

[CMCanavessi]

Even if Gen 10 "forgot" a lot of endgame knowledge compared to the previous networks, it's still (in general) much stronger than Gen 8 (I didn't test Gen 9).

In the last gauntlet I ran, it got 58.5 points out of 92 while Gen 8 had gotten 53.5. But that's not the whole picture, it has now starting to beat engines that in the previous generations it wasn't even drawing with. Maybe it decided that endgame wasn't that important as it could still mate, albeit in more moves...?

Here's the progression in the score of my gauntlet, all of them vs the same engines (an average of around 600 elo):

Gen 6:    40/92
Gen 8:  53.5/92
Gen 10: 58.5/92

And the calculated ratings (with ordo) tell a similar picture, Gen 10 is ~+70 to Gen 8, even if it's worse in the endgame:

 182 Hanzo the Razor x32                    :   989.8     100   53   44    3    75    44   653.4    25    25.0
 183 MFChess 1.3 x32                        :   965.0     100   58   30   12    73    30   654.4    25    25.0
 184 Youk V1.05 x32                         :   934.6     192   66   28   98    42    15  1020.7    47    46.6
 185 StrategicDeep 1.25 x32                 :   922.8      92    7    4   81    10     4  1418.0    23    23.0
 186 Hippocampe v0.4.2 x32                  :   894.0     150   98   18   34    71    12   588.9    15    15.0
 187 Leela Chess Zero Gen 10 x64            :   861.0      92   53   11   28    64    12   655.4    23    23.0
 188 Zoe 0.1 x32                            :   819.4     100   46   29   25    61    29   660.3    25    25.0
 189 Pyotr Amateur Edition v0.6 x32         :   813.7     100   45   30   25    60    30   660.5    25    25.0
 190 Leela Chess Zero Gen 8 x64             :   792.2      92   45   17   30    58    18   655.4    23    23.0
 191 NSVChess 0.14 x32                      :   775.4     250  118   70   62    61    28   622.9    31    26.7
 192 Dikabi v0.4209 x32                     :   739.3     100   23   61   16    54    61   663.5    25    25.0
 193 Easy Peasy 1.0 x32                     :   666.9     250  118   30  102    53    12   629.0    31    26.7
 194 Pyotr Novice Edition v2.6 x32          :   650.8     100   35   22   43    46    22   667.0    25    25.0
 195 Leela Chess Zero Gen 6 x64             :   597.8      92   31   18   43    43    20   655.4    23    23.0
 196 Acqua ver. 20160918 x32                :   509.5     250   95   17  138    41     7   637.8    31    26.7
 197 N.E.G. 1.2 x32                         :   509.5     250   89   29  132    41    12   637.8    31    26.7
 198 Ram 2.0 x32                            :   387.8     250   58   46  146    32    18   644.6    31    26.7
 199 Leela Chess Zero Gen 4 x64             :   369.7     150   43   18   89    35    12   623.9    15    15.0
 200 CPP1 0.1038 x32                        :   326.2     250   45   50  155    28    20   648.1    31    26.7
 201 LaMoSca v0.10 x32                      :   253.0     250    2  111  137    23    44   652.2    31    26.7
 202 POS v1.20 x32                          :   143.9     250   18   45  187    16    18   658.3    31    26.7
 203 EtherealRandom (8.97) x64              :    51.8     100    2   15   83    10    15   691.0    25    25.0
 204 EtherTrueRand 9.21 x64                 :    34.8     250    2   48  200    10    19   664.4    31    26.7
 205 Teki Random Mover x64                  :     0.0     250    0   44  206     9    18   666.3    31    26.7

[Uriopass]

The problem is that Gen10 looks slightly worse than Gen9, maybe try to add Gen9 in your tournament ?

[CMCanavessi]

Yes, but that might be misleading. It looks worse "in self-play", but it may not translate in real play vs other engines. Maybe I'll test Gen 9 later, i can't right now.

[Error323]

I should note however, that I also set move_count to 0. So this input plane will no longer be holding interesting information.

Quoting myself here for reference :sweat_smile: . Anyway I just realized that the chunkparser V1 was already clamping the movecount to stay within a byte (254 for some reason). ~So if the currently training net is performing better, it'll be caused by the bigger windowsize~ this is not true persee, I'm now excluding it basically by setting it to 0. I should maybe also for reference train a net containing only the gen9 data...? I'm not sure what's going on, it seems to me that just using enough samples of the latest net to train the next should always beat it.

[Uriopass]

Yes but the goal is not to beat the previous net, it's also to generalize well against every bot. Or else it might just create an anti-bot and collapse.

[Error323]

Agreed, of course. But what happened with gen10 is that it became worse when training on the previous 3 nets and training on the previous 2... That is, it became worse against gen9, not sure about generations before that though.

[jkiliani]

Do you have two machines you can use for training at the same time? It would certainly be a good experiment to compare two nets trained on the same recent data, but different numbers of older training games...

[killerducky]

Is gen10 = Id14 = 8f60? Maybe we can switch to referring to them as ID## or gen10_8f6? It's a bit confusing.

We had 100% improvements over the previous gen before, maybe it's not unexpected that some will show regression in terms of a match vs previous. Too bad we can't really compare to LZ because the beginning was bugged. Current LZ fails matches very often but it's very different since the level is already very high, and the training method is quite different as well.

[Error323]

Once V2 data is the only data I can run two training instances on both GPUs at the same time.

[Error323]

@killerducky yes sorry, I'll start referring to their sha256 hashes from now on. gen10 is indeed
8f60220e.

[Uriopass]

Why not convert all the V1 data to V2 data then ? Or is it too hard ?

[Error323]

It's not hard, but not trivial either. And there is a lot of work todo for automation. I only have so much time :(

[Uriopass]

Also how's the training going ? Since there are so many datapoints can it converge ?

[Error323]

It's still in 0.04 lr phase. It's gonna switch to 0.002 at the 35K mark.

[jkiliani]

So we're just about to reach the learning rate change... it will probably be finished sometime after midnight unless I miss my guess on training speed?

[Error323]

Mja, and I wanna sleep soon. But before I goto bed I'll run a small benchmark of 30 games against gen9=
5c8d14d5. And let you know :-)

[killerducky]

What's with the cycling up/down in the test data for the MSE Loss chart? It looks similar to when the training data of different generations was not shuffled properly.

[Error323]

That's correct. It's not properly shuffled, I have fixed it in https://github.com/Error323/leela-chess/commit/33ce2fccce1da7a00f5f97fb4d8278ae2deb14a4.

[jjoshua2]

I thought of a use for move count. It is a quick indicator of game phase. Opening, middlegame, or endgame. SF uses different weights for mid and endgame and phases from one to another, so it gains elo. This could be probably be more accurately determined with nn using piece counts, but might take an extra layer, but piece counts isn't even given as an input right? So would take more layers to count material, and then do something with it.

[Uriopass]

I think for the opening it's pretty easy to check since all the "previous moves" planes are set to zero so it's trivial to determine which move you are for the first 8 moves for a nn.

[Error323]

Mmm I don't know. It's pure speculation, we have no idea how the net would use it. We do know it's redundant. The net can deduce it from other intel, and since it was clamped to 254 from day 1. It's flawed.

[jkiliani]

I think that it's completely unnecessary and even counterproductive to use move count for classifying game stage into opening, midgame, endgame. The net doesn't need anything more than the number of pieces left to determine that for itself very accurately. And piece count is an indirect input, which a neural net should have no trouble determining simply from the board position.

[jjoshua2]

After 500 ply it is pretty solidly endgame, so it should be fully using endgame values by then, but that doesn't mean clamping it at 254 is optimal. See SF example. We don't know if it helps, since its hard to see into a NN, but it seems a reasonable theory that it could. The net should still be able to be way better than its current poor level even without separate mid/end values though.

@jkiliani I think piece count would be better too, but I don't think it is an input, and with only 5 layers that doesn't leave much space for it to be calculated and then used.

[Error323]

I removed tensorboard's smoothing so you can really see how close the testset is to the trainset on the loss.

Score of f393628a vs 5c8d14d5: 36 - 13 - 1  [0.730] 50
Elo difference: 172.78 +/- 113.19

We are back in business! :rocket:

[jkiliani]

So you're testing the fully trained net against this one tomorrow morning?

[Error323]

I just uploaded this one, because why not. And I'll now run a new training with 100K window.

[JackThomson2]

Great work good to see we're back on track

[evalon32]

Network | KQQvK | KQRvK | KRRvK | KQvK | KRvK | KBBvK | KBNvK
--- | --- | --- | --- | --- | --- | --- | ---
gen11 | 100 / 7.5 | 97 / 11.0 | 99 / 8.5 | 97 / 21.3| 58 / 23.1 | 0 / inf | 0 / inf
gen12 | 100 / 6.6 | 100 / 13.9 | 100 / 10.9 | 94 / 17.5 | 76 / 26.7 | 13 / 33.7 | 0 / inf

EDIT: Added gen12

%Win vs Self

%Win vs Stockfish

[Uriopass]

The % win against SF is a very good idea, I assume you took the "perfect" stockfish ?
EDIT : Looks like the KRvR is missing from the stockfish data ? Or was it 0 everywhere ?

[jkiliani]

I think it must have been zero everywhere, along with KBBvK and KBNvK. But agreed that this is not evident from the plot. Winning KRvK against a perfect opponent requires the basic idea of how to get the other king to the edge, which it seems the current nets don't have yet.

[mooskagh]

Why is QQK vs K 100% win when played random against stockfish, while random vs random it's less?

[jkiliani]

That's a very good question...

[sf-x]

Why is QQK vs K 100% win when played random against stockfish, while random vs random it's less?

Because you confused the graphs. They show exactly the opposite.

[mooskagh]

Hm, I don't see where I'm wrong, I cannot see how the graphs can be interpreted differently.

[sf-x]

sigh
The top graph clearly shows that lc0 can convert ~100% of KQQvK against self. Bottom one is lc0 v. SF, shows ~94% (for gen11).

[mooskagh]

I was talking about random play (gen0)

[sf-x]

Ah. The "programmed stupidity": SF, when it sees a forced mate for itself, tries to delay it as much as possible, rather than trying to trick the opponent.

[evalon32]

@Uriopass I did not use an EGTB with Stockfish, but I don't think it mattered for now, because it can see far enough for perfect play against Q or more, and the scores with less than Q were all 0 anyway.

@Uriopass, @jkiliani Correct, all networks scored 0% on KRvK, KBBvK, and KBNvK. Even the SL-trained network (kbb1-64x6-796000, not shown in the graph) scored 0%.

@mooskagh, @sf-x You noticed, huh :) This is an artifact of a subtle unintentional difference in inputs to which the RNG network is extremely sensitive. Here's what happens: For KQQvK, I am using FEN "8/3k4/8/8/8/8/3KQQ2/8 w - - 0 10" as the starting position. It turns out that (a) cutechess-cli (which I used for stockfish matches, but not for selfplay) overrides the move number, changing the starting position to "8/3k4/8/8/8/8/3KQQ2/8 w - - 0 1" and (b) LC with RNG reliably finds a mate in two from "8/3k4/8/8/8/8/3KQQ2/8 w - - 0 1", but not from "8/3k4/8/8/8/8/3KQQ2/8 w - - 0 10". If I start selfplay from "8/3k4/8/8/8/8/3KQQ2/8 w - - 0 1", it gets the same 100% score as vs Stockfish.

[killerducky]

Is the last number in the FEN code the move number? @Error323 are you setting move number to zero in training? But during play it's non zero? So the network will be confused by that input. Seems like a problem.

[evalon32]

@killerducky Yes, that's the move number.

[Error323]

Indeed I'll fix it asap @killerducky. Thx

[evalon32]

Network | KQQvK | KQRvK | KRRvK | KQvK | KRvK | KBBvK | KBNvK
--- | --- | --- | --- | --- | --- | --- | ---
id18 | 100 / 8.7 | 100 / 11.8 | 100 / 10.9 | 100 / 10.4 | 47 / 10.5 | 0 / inf | 0 / inf
id19 | 100 / 9.1 | 100 / 11.1 | 100 / 10.9 | 100 / 10.8 | 64 / 30.4 | 0 / inf | 0 / inf
id20 | 100 / 8.7 | 100 / 10.5 | 100 / 9.3 | 100 / 10.5 | 79 / 29.2 | 1 / 14.0 | 0 / inf
id22 | 100 / 7.6 | 100 / 10.1 | 100 / 11.0 | 100 / 10.0 | 71 / 27.4 | 0 / inf | 0 / inf
id23 | 100 / 11.1 | 100 / 12.6 | 100 / 11.6 | 100 / 10.4 | 58 / 21.2 | 0 / inf | 0 / inf
id26 | 100 / 10.4 | 100 / 11.3 | 100 / 12.0 | 100 / 10.5 | 83 / 24.0 | 0 / inf | 0 / inf
id27 | 100 / 10.3 | 100 / 13.3 | 100 / 12.7 | 100 / 10.1 | 88 / 28.3 | 1 / 11.0 | 1 / 45.0
id29 | 100 / 11.0 | 100 / 13.1 | 100 / 13.7 | 100 / 10.8 | 92 / 27.2 | 1 / 19.0 | 0 / inf
id30 | 100 / 10.1 | 100 / 11.8 | 100 / 12.8 | 100 / 11.5 | 98 / 31.9 | 3 / 35.0 | 0 / inf
id31 | 100 / 10.9 | 100 / 13.2 | 100 / 13.6 | 100 / 11.2 | 100 / 25.0 | 1 / 32.0 | 0 / inf
id33 | 100 / 8.3 | 100 / 11.4 | 100 / 13.5 | 100 / 10.7 | 99 / 25.7 | 3 / 30.0 | 0 / inf
id37 | 100 / 9.8 | 100 / 12.6 | 100 / 12.2 | 100 / 11.6 | 100 / 23.7 | 1 / 21.0 | 2 / 23.0

Solid progress, including a breakthrough in KRvK vs Stockfish. Also, we've now surpassed the kbb1 network in all categories! Here are the results from head-to-head between kbb1 and id19, 20 games per side per variant.

Sides | KQQvK | KQRvK | KRRvK | KQvK | KRvK | KBBvK | KBNvK
--- | --- | --- | --- | --- | --- | --- | ---
id19 - kbb1 | 20 | 20 | 20 | 20 | 20 | 2 | 1
kbb1 - id19 | 20 | 20 | 17 | 0 | 1 | 0 | 0

Here we may be seeing the effect that @sf-x mentioned: with KQvK, kbb1 can mate Stockfish ~half the time, but couldn't mate id19 even once!

Disclaimer: some of the old stats, especially with very early networks, would be different with current code (after fcd67bd5a4d34c4fceaa85ccf2264c33cf235ae0). I'm not yet sure what to do about that. When I started this, I considered generating random legal positions instead of an arbitrary single one. That might have been a better metric, but I was too lazy :)

EDIT: Added id20 through id98.
EDIT: No longer tracking major piece variants (all of them are solid 100% now).

%Win vs Self

%Win vs Stockfish

[glinscott]

Interesting that the network surpassed the kbb1 one in the endgames, but is still quite a bit weaker in match play.

[jkiliani]

@evalon32 Could you add id27 and id29? I think the policy priors of id29 continued to improve so it must have regressed at something for it to get the even result against id27, could it be the endgame?

[evalon32]

@glinscott Interesting, but not unexpected: kbb1 was trained on games that tend to be won without a checkmate.
@jkiliani Done, and it doesn't appear to be the endgame. Slow but steady progress here.

[CMCanavessi]

Only BB and BN remain to learn, it has mastered the rest already.

[jkiliani]

Cool, looks like it may be just starting to understand how to mate with two bishops. Bishop+knight I personally find really hard, I know I couldn't do it against Stockfish.

Another idea you could try is to make a sample of some endgame positions including pawns that are theoretically won for white (classical example would be the Lucena position, or even KPvK) and evaluate if the nets can win from those positions.

I would be really interested if current Leela can win the Lucena position against Stockfish, since it's not easy if you haven't studied endgame.

[evalon32]

@jkiliani Both KBBvK and (amazingly!) KBNvK appear to be taking off.

I did some ad-hoc testing of KPvK positions about a week ago, and Leela pretty much aced them already then (although, I was using 100K playouts, not 800). That includes a position like this. Leela doesn't realize that this is drawn until the very end, but has no problem drawing as black, and will try to swindle out a win as white.
Lucena was interesting... Leela won against Stockfish, but only because Stockfish gave up too early. The game went:

[TimeControl "180+1"]
[FEN "1K1k4/1P6/8/8/8/8/r7/2R5 w - - 0 1"]

1. Rd1+ Ke7 2. Kc8 Rc2+ 3. Kb8 Ra2 4. Rd4 Ke6 5. Kc8 Rc2+ 6. Kb8 Ra2 7. Kc7 Rc2+ 8. Kb6 Rg2 9. Rb4 Rg8 10. Kc7 Kf5 11. b8=Q

Yet after 8...Rb1+, Leela can't find a winning line even with 10M playouts.

[evalon32]

Update on the Lucena position: As of ID83, Leela finds the correct followup:
https://lichess.org/SWlxBXQE#0

[CMCanavessi]

Weird how after ID 83 it "forgot" a lot of endgame knowledge which is just now starting to get back...

[zz4032]

Do you think that high randomness on moves ("temperature" setting) hurts LCZero on improving in the endgame? The idea was mentioned on talkchess forum some time ago. Maybe randomness should better decrease step by step over the course of a (training) game?

[evalon32]

Going forward, I'll be updating this spreadsheet (charts included):
https://docs.google.com/spreadsheets/d/1uY7fplZzeXi8H52LK0L6Do2oYgF1v5VW3K_AjNS8l0M/edit?usp=sharing