Leela-chess: Low prior for winning move when history is on

Created on 10 May 2018 · 22Comments · Source: glinscott/leela-chess

(discussed on discord, originally reported here:
https://groups.google.com/d/topic/lczero/iiOlKkuhCGw/discussion
)

In this position

position startpos moves c2c4 e7e5 g2g3 g8f6 f1g2 d7d5 c4d5 f6d5 b1c3 d5b6 g1f3 b8c6 a2a3 c8e6 d2d3 f8e7 e1g1 e8g8 b2b4 a7a5 b4b5 c6d4 f3e5 e7f6 f2f4 f6e5 f4e5 d4b3 c1e3 b3a1 d1a1 b6d5 e3d4 d5c3 a1c3 e6d5 e2e4 d5e6 d4c5 f8e8 c5e3 c7c5 b5c6 b7c6 d3d4 d8d7 d4d5 c6d5 e4d5 e6d5 f1d1 d5g2

Prior for move Rxd7 (rook takes hanging queen) is 0.03% if history planes are filled, and 79.11% if history planes are empty (e.g. when the position is initialized with fen).

It seems that it "sees" bishop at d5 in history and thinks that Rxd7 is illegal move.

When testing on previous networks:

id130:  P(Rxd7) = 97.85%
id210:  P(Rxd7) = 96.65%
id240:  P(Rxd7) = 97.19%
id242:  P(Rxd7) = 96.98%
id252:  P(Rxd7) = 93.07%
id266:  P(Rxd7) = 0.01%

Important!

Numbers above are of lc0, for lczero it went worse few networks earlier!

Source

mooskagh

👍4

Most helpful comment

I'm just reading code, but it looks wrong. input_data is filled with too many planes.bit[c][i]. I guess the last few of these planes will just be uninitialized memory (~~read: garbage~~ it's guaranteed to be cleared to all zeros). Then rule50_count, move_count, and all 1s are appended. From there I guess opencl.forward will use ~~the garbage inputs~~ the all zeros instead of the real ones for those 3.

I think this code:

    input_data.reserve(MAX_INPUT_CHANNELS * width * height);
    for (int c = 0; c < MAX_INPUT_CHANNELS - 3; ++c) {

Should be this:

    input_data.reserve(get_input_channels() * width * height);
    for (int c = 0; c < get_input_channels() - 3; ++c) {

killerducky on 11 May 2018

👍4

All 22 comments

It seems to me that something changed in lczero that made it generate corrupt training data.

That would explain why this issue appeared first in lczero and only few networks later (when training goes further) in lc0.

mooskagh on 10 May 2018

The planes for this move for lc0 is here: https://gist.github.com/mooskagh/35f38de8ceeb3dea7fdab8c6bc639d5f
if someone wants to compare.
Planes for lczero: https://gist.github.com/mooskagh/4d99a0dac8a3aae26054406829599947

At network id252, it returns prior of 93.07% for move Rxd7 (move idx =95)

mooskagh on 10 May 2018

For lczero (same for all of 0.7, 0.8, and 0.9 engines), probability from policy for Rxd7 with full history:

ID234: 98.0%
ID235: 97.8%
ID236: 97.6%
ID237: 97.8%
ID238: 97.7%
ID239: 97.1%
ID240: 96.5%
ID241: 96.2%
ID242: 91.7%
ID243: 17.8%
ID244: 14.39%
ID245: 0.83%

EDIT: Added several previous nets for more context.

apleasantillusion on 10 May 2018

👍3

In lczero code:

Network::Netresult Network::get_scored_moves_internal(
    const BoardHistory& pos, NNPlanes& planes, DebugRawData* debug_data) {
  assert(MAX_INPUT_CHANNELS == planes.bit.size() + 3);
 // MAX_INPUT_CHANNELS == 120 !!!! Should be 112.
  constexpr int width = 8;
  constexpr int height = 8;

UPD: Actually, in this particular place, 120 is expected.

So what seems to happen is:
rule50_count and pad-with-ones are not planes 109 and 111, but. "117 and 119"
They don't exist, so they corrupt other planes.
So when we started to fill last plane with ones, we just started to corrupt other planes more actively, possibly overwriting vital parts.

Or maybe somewhere we flipped v2 to v3.
And somewhere in the code (if version == 2, use 112 planes)

mooskagh on 10 May 2018

👍4

On a second look, the problem doesn't seem to be as serious.
It seems that 50-move-rule plane and padding plane are indeed 117 and 119, but they are not used later..

That doesn't explain sudden regression though, so something else must be a reason.

mooskagh on 10 May 2018

👍1

Even 91% seems very low for a plain queen capture. What were the values for versions preceding this?

ASilver on 10 May 2018

👍1

Looking at the dumps for the planes.. lc0 has 112 planes.. planes higher than 112 should be ignored for lczero... but the difference is plane 112.. it should be all 1s, but it isnt for lczero?

gyathaar on 11 May 2018

👍1

I think this code:

    input_data.reserve(MAX_INPUT_CHANNELS * width * height);
    for (int c = 0; c < MAX_INPUT_CHANNELS - 3; ++c) {

Should be this:

    input_data.reserve(get_input_channels() * width * height);
    for (int c = 0; c < get_input_channels() - 3; ++c) {

killerducky on 11 May 2018

👍4

yes.. seems like lczero puts the 1s in plane 120.. which is then thrown away...

gyathaar on 11 May 2018

👍1

To follow up - I agree with killerduckys proposed fix. But the last few planes are probably all 0, rather than uninitialized random 0/1s (due to bitset default constructor being zeroing).

Tilps on 11 May 2018

👍1

I tested a 20x256 net I have been training with lczero v.9, of which the current window of games are all greater than 11,200,000, which has a sampling of approximately 100 positions per game, at a lr of .01, with a window of 500,000 games. It finds d1d7, so it appears the problem might be related to oversampling of games or simply my net isn’t capable of reproducing the bug at a lr of .01, which doesn’t produce a very strong net, with post v.8 data.

https://pastebin.com/wFrLVm50
weights file:
https://drive.google.com/open?id=1zW1527W-IfwvjNXZ9mSOMJeYVblUesU-

go nodes 1
info string Rxd7 -> 2 (V: 80.29%) (N: 98.56%) PV: Rxd7 Bh3 Rd6
info string stm White winrate 75.41%

go nodes 10000
info string Rxd7 -> 5001 (V: 82.93%) (N: 98.56%) PV: Rxd7 Bh3 Ra7 h6 Rxa8 Rxa8 Qc6 Rb8 Qe4 Be6 Bd2 Rd8 Bxa5 Rd1+ Kf2
info string stm White winrate 82.91%

theseapples on 11 May 2018

Could you quickly check that the fix helps in that particular position with network id252?

UPD: Checked myself, and confirm that with the fix it for id252 it returns P=93.07% for Rxd7 in lczero.

mooskagh on 11 May 2018

Also what's really good, is that outputs from neural net are very similar for lczero and lc0, up to a very-very high precision! (Noise/differences in backed computation were blamed for that before).

mooskagh on 11 May 2018

Let's use this issue to track how lczero behaves on the same position in future nets, to see how quickly it recovers.

mooskagh on 11 May 2018

I double checked.. the model training code appears to have been training the neural network with the correct inputs the hole time
so I think the impact will be minimal.. it is just that it lately have been trained with "badly" played games..(and is quite bad at estimating the badly played games because it doesn't have the same inputs) with the lczero playing code correct, the performance should go up quite quickly I think when the fixed games start entering the training data

gyathaar on 11 May 2018

It's still a mystery why that started to happen with v0.8.
We should take a network few generations before bad one, and check how counters look for that Rxd7 moves in v0.7 vs v0.8 (in like-training environment, e.g. with noise and temperature).

mooskagh on 11 May 2018

info string  Rxd7 ->     386   (V: 85.32%) (N: 18.80%) PV: Rxd7 Bh3 Rc7 h6 e6 Bxe6 Bd4 Rad8 Bxg7 Rd1+

This is with lczero, with an unfixed binary, and Id 277.

jkiliani on 11 May 2018

Disregard post (deleted) on possible issue with move first reported in forum. I tested and cannot reproduce the problem.

ASilver on 11 May 2018

./lczero -w weights/id279-91f1259a 
Using 2 thread(s).
Detecting residual layers...v2...192 channels...15 blocks.
BLAS Core: Sandybridge
info string Found 0 wdl tablebases
info string Found 0 dtz tablebases
position startpos moves c2c4 e7e5 g2g3 g8f6 f1g2 d7d5 c4d5 f6d5 b1c3 d5b6 g1f3 b8c6 a2a3 c8e6 d2d3 f8e7 e1g1 e8g8 b2b4 a7a5 b4b5 c6d4 f3e5 e7f6 f2f4 f6e5 f4e5 d4b3 c1e3 b3a1 d1a1 b6d5 e3d4 d5c3 a1c3 e6d5 e2e4 d5e6 d4c5 f8e8 c5e3 c7c5 b5c6 b7c6 d3d4 d8d7 d4d5 c6d5 e4d5 e6d5 f1d1 d5g2
go nodes 1
update_root, 0 -> 0 expanded nodes (0.0% reused)
NNCache: 0/0 hits/lookups = 0.0% hitrate, 0 inserts, 0 size
info depth 6 nodes 1 nps 0 tbhits 0 score cp -429 time 259 pv d1d7

info string   Bc5 ->       0   (V: 18.49%) (N:  0.22%) PV: Bc5 
info string   Ba7 ->       0   (V: 18.49%) (N:  0.22%) PV: Ba7 
info string   Rf1 ->       0   (V: 18.49%) (N:  0.23%) PV: Rf1 
info string   Qc8 ->       0   (V: 18.49%) (N:  0.25%) PV: Qc8 
info string   Rd6 ->       0   (V: 18.49%) (N:  0.28%) PV: Rd6 
info string    g4 ->       0   (V: 18.49%) (N:  0.28%) PV: g4 
info string   Rb1 ->       0   (V: 18.49%) (N:  0.28%) PV: Rb1 
info string   Ra1 ->       0   (V: 18.49%) (N:  0.28%) PV: Ra1 
info string   Bb6 ->       0   (V: 18.49%) (N:  0.29%) PV: Bb6 
info string   Bg5 ->       0   (V: 18.49%) (N:  0.29%) PV: Bg5 
info string   Re1 ->       0   (V: 18.49%) (N:  0.31%) PV: Re1 
info string   Bd2 ->       0   (V: 18.49%) (N:  0.33%) PV: Bd2 
info string    h3 ->       0   (V: 18.49%) (N:  0.33%) PV: h3 
info string   Bf4 ->       0   (V: 18.49%) (N:  0.34%) PV: Bf4 
info string   Kf2 ->       0   (V: 18.49%) (N:  0.34%) PV: Kf2 
info string   Qc7 ->       0   (V: 18.49%) (N:  0.36%) PV: Qc7 
info string   Rd5 ->       0   (V: 18.49%) (N:  0.37%) PV: Rd5 
info string  Qxa5 ->       0   (V: 18.49%) (N:  0.38%) PV: Qxa5 
info string   Bf2 ->       0   (V: 18.49%) (N:  0.39%) PV: Bf2 
info string   Qc6 ->       0   (V: 18.49%) (N:  0.40%) PV: Qc6 
info string    a4 ->       0   (V: 18.49%) (N:  0.40%) PV: a4 
info string    h4 ->       0   (V: 18.49%) (N:  0.42%) PV: h4 
info string   Bh6 ->       0   (V: 18.49%) (N:  0.48%) PV: Bh6 
info string   Qc5 ->       0   (V: 18.49%) (N:  0.51%) PV: Qc5 
info string   Bc1 ->       0   (V: 18.49%) (N:  0.57%) PV: Bc1 
info string   Rc1 ->       0   (V: 18.49%) (N:  0.57%) PV: Rc1 
info string   Qb4 ->       0   (V: 18.49%) (N:  0.66%) PV: Qb4 
info string   Qb3 ->       0   (V: 18.49%) (N:  0.66%) PV: Qb3 
info string   Qa1 ->       0   (V: 18.49%) (N:  0.67%) PV: Qa1 
info string   Qb2 ->       0   (V: 18.49%) (N:  0.79%) PV: Qb2 
info string   Bd4 ->       0   (V: 18.49%) (N:  0.95%) PV: Bd4 
info string   Qc1 ->       0   (V: 18.49%) (N:  0.96%) PV: Qc1 
info string   Qd3 ->       0   (V: 18.49%) (N:  1.06%) PV: Qd3 
info string   Qc4 ->       0   (V: 18.49%) (N:  1.12%) PV: Qc4 
info string   Qe1 ->       0   (V: 18.49%) (N:  1.17%) PV: Qe1 
info string   Qd2 ->       0   (V: 18.49%) (N:  1.28%) PV: Qd2 
info string   Rd3 ->       0   (V: 18.49%) (N:  1.40%) PV: Rd3 
info string   Qc2 ->       0   (V: 18.49%) (N:  2.84%) PV: Qc2 
info string   Rd2 ->       0   (V: 18.49%) (N:  3.10%) PV: Rd2 
info string    e6 ->       0   (V: 18.49%) (N:  3.30%) PV: e6 
info string   Qd4 ->       0   (V: 18.49%) (N:  3.65%) PV: Qd4 
info string   Rd4 ->       0   (V: 18.49%) (N:  4.85%) PV: Rd4 
info string  Kxg2 ->       0   (V: 18.49%) (N: 28.29%) PV: Kxg2 
info string  Rxd7 ->       1   (V: 85.94%) (N: 34.10%) PV: Rxd7 Bh3
info string stm White winrate 52.22%

info depth 7 nodes 2 nps 2 tbhits 0 score cp 19 time 444 pv d1d7 g2h3
bestmove d1d7

Compiled from current next

dubslow on 12 May 2018

info string Rxd7 -> 1 (V: 85.94%) (N: 34.10%) PV: Rxd7

That shows how off-center the NN is. 86% is a really bad result. Not as bad as not playing it at all of course, but much too low a score. 100% might be ideal, but upper 90s should be the least acceptable score.

ASilver on 12 May 2018

Anything that's a clear majority is fine, it doesn't matter if it's 92 or 97.

Btw id280 had the prior back up to 89.5% after 279's 34.1%

dubslow on 12 May 2018

I think this issue, at least, can be officially closed (can't speak for most other blunders in the other thread). With normal history (position x-plies-no-filler-moves), id285 just produced the following policy:

info string   Ba7 ->       0   (V: 58.28%) (N:  0.02%) PV: Ba7 
info string   Qc8 ->       0   (V: 58.28%) (N:  0.02%) PV: Qc8 
info string   Rf1 ->       0   (V: 58.28%) (N:  0.02%) PV: Rf1 
info string   Kf2 ->       0   (V: 58.28%) (N:  0.02%) PV: Kf2 
info string    h3 ->       0   (V: 58.28%) (N:  0.03%) PV: h3 
info string   Bb6 ->       0   (V: 58.28%) (N:  0.03%) PV: Bb6 
info string    g4 ->       0   (V: 58.28%) (N:  0.03%) PV: g4 
info string   Bg5 ->       0   (V: 58.28%) (N:  0.03%) PV: Bg5 
info string   Qc6 ->       0   (V: 58.28%) (N:  0.03%) PV: Qc6 
info string   Bc5 ->       0   (V: 58.28%) (N:  0.03%) PV: Bc5 
info string  Qxa5 ->       0   (V: 58.28%) (N:  0.03%) PV: Qxa5 
info string   Bd4 ->       0   (V: 58.28%) (N:  0.03%) PV: Bd4 
info string   Re1 ->       0   (V: 58.28%) (N:  0.03%) PV: Re1 
info string   Qc7 ->       0   (V: 58.28%) (N:  0.03%) PV: Qc7 
info string    a4 ->       0   (V: 58.28%) (N:  0.03%) PV: a4 
info string   Qc5 ->       0   (V: 58.28%) (N:  0.04%) PV: Qc5 
info string   Bf4 ->       0   (V: 58.28%) (N:  0.04%) PV: Bf4 
info string    h4 ->       0   (V: 58.28%) (N:  0.04%) PV: h4 
info string   Rb1 ->       0   (V: 58.28%) (N:  0.04%) PV: Rb1 
info string   Ra1 ->       0   (V: 58.28%) (N:  0.04%) PV: Ra1 
info string   Bh6 ->       0   (V: 58.28%) (N:  0.04%) PV: Bh6 
info string   Qb4 ->       0   (V: 58.28%) (N:  0.04%) PV: Qb4 
info string   Bd2 ->       0   (V: 58.28%) (N:  0.04%) PV: Bd2 
info string   Qa1 ->       0   (V: 58.28%) (N:  0.05%) PV: Qa1 
info string   Qb2 ->       0   (V: 58.28%) (N:  0.05%) PV: Qb2 
info string   Bf2 ->       0   (V: 58.28%) (N:  0.05%) PV: Bf2 
info string   Qd4 ->       0   (V: 58.28%) (N:  0.05%) PV: Qd4 
info string   Bc1 ->       0   (V: 58.28%) (N:  0.05%) PV: Bc1 
info string   Qb3 ->       0   (V: 58.28%) (N:  0.06%) PV: Qb3 
info string   Rc1 ->       0   (V: 58.28%) (N:  0.06%) PV: Rc1 
info string   Qd3 ->       0   (V: 58.28%) (N:  0.06%) PV: Qd3 
info string   Rd5 ->       0   (V: 58.28%) (N:  0.07%) PV: Rd5 
info string   Qc4 ->       0   (V: 58.28%) (N:  0.07%) PV: Qc4 
info string   Qd2 ->       0   (V: 58.28%) (N:  0.07%) PV: Qd2 
info string   Qe1 ->       0   (V: 58.28%) (N:  0.08%) PV: Qe1 
info string   Rd3 ->       0   (V: 58.28%) (N:  0.09%) PV: Rd3 
info string   Qc1 ->       0   (V: 58.28%) (N:  0.09%) PV: Qc1 
info string   Rd6 ->       0   (V: 58.28%) (N:  0.10%) PV: Rd6 
info string   Rd4 ->       0   (V: 58.28%) (N:  0.11%) PV: Rd4 
info string    e6 ->       0   (V: 58.28%) (N:  0.14%) PV: e6 
info string   Rd2 ->       0   (V: 58.28%) (N:  0.16%) PV: Rd2 
info string   Qc2 ->       0   (V: 58.28%) (N:  0.20%) PV: Qc2 
info string  Kxg2 ->       0   (V: 58.28%) (N:  0.42%) PV: Kxg2 
info string  Rxd7 ->       1   (V: 87.21%) (N: 97.25%) PV: Rxd7 Bh3
info string stm White winrate 72.75%

dubslow on 13 May 2018

Was this page helpful?

0 / 5 - 0 ratings