Leela-chess: Is it possible to train the value output in a supervised (or even self-play) manner as AZ paper explains?

[Zeta36]

I've been working for a while in this kind of AZ projects. I started even this project some months ago: https://github.com/Zeta36/chess-alpha-zero

And I have a doubt I'd like to ask you. It's about the training of the value output. If I backprop the value output always with an integer (-1, 0, or 1) the NN should quickly be stuck in a local minimal function ignoring the input and always returning the mean of this 3 values (in this case 0). I mean, as soon as the NN learn to return always near 0 values ignoring the input planes, there will be no more improvements since it will have a high accuracy value (>25%) almost intermediately after some steps.

In fact I did a toy experiment to confirm this. As I mention the NN was unable to improve after reaching 33% accuracy (~0.65 loss in mean square). And this has sense if the NN is always returning 0 (very near zero values). Imagine we introduce a dataset of 150 games: ~50 are -1, ~50 are 0 and ~50 are 1. If the NN learns to say always near 0, we get an instant loss of (mse): 100/150 ~ 0.66 and an accuracy of ~33% (1/3).

How the hell did DeepMind to train the value network with just 3 integer values to backpropagate??
I thought the tournament selection (the evaluation worker) was implicated in helping to overcome this local minimum (stabilizing the training), but in its last paper they say they removed the eval process (??)...so, I don't really know what to think.

I don't know neither if the self-play can help in this issue. In last term we are still back-propagating an integer from a domain of just 3 values.

Btw, you can see in our project in https://github.com/Zeta36/chess-alpha-zero we got some "good" results (in a supervised way) but I suspect it was all thanks to the policy network guiding the MCTS exploration (with a value function returning always near 0 values).

What do you think about this?

[glinscott]

Hi @Zeta36! Yes, getting the network away from draws does seem to be a challenge. First step is to validate that with supervised training we can get to a reasonable strength, and the search is not making any huge blunders. Then potentially doing some self-play with that network and seeing if it can learn to improve itself.

[Zeta36]

DeepMind in the paper does not say they get rid of the draws. Moreover, they say they count as a draw any game taking more than _n_ number of movements (n>=average movements in a chess game) so the bias for the NN stuck in a local minimum saying always 0 would be even greater.

I don't know it's all a very strange.

[Error323]

I do think the dual head helps, both the probability head and the value head are backpropagated. What happens when you remove the draw games?

[Zeta36]

But the dual head ends the value part with an independent FC layer than can be easily backprop to have weights near to zero so the head would return always 0 at the same time that would not affect too much the rest of the common network. I'm not so sure the dual head can solve the problem.

[glinscott]

The network does appear to (slowly) be getting better from the supervised learning. This happened after I dropped the learning rate from 0.05 to 0.005.

Latest results:

step 576000, policy loss=2.67612 mse=0.0616492 reg=0.400642 (3618.78 pos/s)
step 576000, training accuracy=28.0371%, mse=0.0602521

[Error323]

@glinscott can you examine the FC layer on the value head in the latest weights.txt? Show some statistics? I agree with @Zeta36 it would indeed be problematic if they converge to 0.

I'll see if I can write out tf.example bytes and learn from that. Good idea.

[Zeta36]

@glinscott you have to focus on the mse loss (the head value part). If I'm correct, the model (the head part) should converge fast (around 0.66, with near 33% of accuracy) and get stuck in returning always (very near) zero values.

[glinscott]

@Zeta36 when playing games, the network is definitely returning values not close to zero. Eg. in the following position, we can see the network is learning that white is winning. Unfortunately, the policy has it playing Ne5, which is a repetition...

1r1n4/1p1P1b2/2p1p1nk/PPP3rp/3PPP2/3N3P/1R2N1B1/3Q1RK1 w - - 7 45
eval: 0.892773
Ne5 0.549349
Rb4 0.069966
fxg5+ 0.046496
Qa4 0.042237
Qc2 0.037499
Nb4 0.031261
Rb3 0.025240

[Zeta36]

@glinscott do you have a summary of your current value head loss (and accuracy)?

[glinscott]

Here are the latest training steps:

step 580100, policy loss=2.64198 mse=0.0594456 reg=0.395409 (2585.08 pos/s)
step 580200, policy loss=2.64212 mse=0.0593968 reg=0.395284 (2826.09 pos/s)
step 580300, policy loss=2.64205 mse=0.0591733 reg=0.395159 (3744.2 pos/s)
step 580400, policy loss=2.64157 mse=0.0593887 reg=0.395034 (3165.17 pos/s)
step 580500, policy loss=2.64044 mse=0.0594478 reg=0.394909 (3733.18 pos/s)
step 580600, policy loss=2.63911 mse=0.0594092 reg=0.394784 (3285.71 pos/s)
step 580700, policy loss=2.63862 mse=0.0591817 reg=0.39466 (3275.92 pos/s)
step 580800, policy loss=2.63818 mse=0.0592512 reg=0.394536 (3754.25 pos/s)
step 580900, policy loss=2.6374 mse=0.0594759 reg=0.394412 (3215.41 pos/s)
step 581000, policy loss=2.63621 mse=0.0593435 reg=0.394289 (3251.14 pos/s)
step 581100, policy loss=2.63555 mse=0.0594918 reg=0.394166 (3780.63 pos/s)
step 581200, policy loss=2.6349 mse=0.0597201 reg=0.394043 (3297.47 pos/s)
step 581300, policy loss=2.63339 mse=0.0596737 reg=0.393921 (3817.32 pos/s)
step 581400, policy loss=2.6337 mse=0.0595412 reg=0.393798 (3266.03 pos/s)
step 581500, policy loss=2.6338 mse=0.0591958 reg=0.393676 (3276.31 pos/s)
step 581600, policy loss=2.63308 mse=0.0587861 reg=0.393554 (3790.01 pos/s)
step 581700, policy loss=2.63232 mse=0.0587501 reg=0.393433 (3224.15 pos/s)
step 581800, policy loss=2.6325 mse=0.0587086 reg=0.393312 (3715.49 pos/s)
step 581900, policy loss=2.6318 mse=0.0589177 reg=0.393191 (3213 pos/s)
step 582000, policy loss=2.6317 mse=0.0587718 reg=0.39307 (3260.37 pos/s)
step 582000, training accuracy=29.0234%, mse=0.0632076

The MSE loss is divided by 4 to match Google's results though.

[Zeta36]

hmmmm...well, maybe I'm wrong but I don't know why I'm wrong. It should get stuck. It's simple statistics. You are using supervised data, aren't you? Do you know if the PGN you are using is biased in some direction (with little or null zero games or something like that)? Also, how many movements are you using for the optimization? If you are using little (or biased) movements the NN could find a way to escape the convergence into 0.

[glinscott]

The pgn is from @gcp, who used SF self-play games (https://sjeng.org/dl/sftrain_clean.pgn.xz). Stats indicate it's a normal sample of high-level chess games.

   6264 [Result "0-1"]
  15382 [Result "1-0"]
  53322 [Result "1/2-1/2"]

[Zeta36]

Mmmm, perhaps it's a normal sample of high-level chess games, but there is an statistical bias in those results. Too much white wins versus black. That could cause the model to escape the fast convergence into zero.

I don't want to bother you too much, but it would be great to check with a 33% (white wins), 33% draws and 33% black wins. Maybe it would be just enough training with the same number of 0-1 and 1-0 results to rule out my position.

And this could be important because with a random initial sel-play model there will be probably same white and black wins in the beginning (what could cause if I'm correct a fast and bad convergence of the value head).

[glinscott]

@Zeta36 agreed, I do want to implement that. But the network at least appears to be learning well with these games so far:

And the latest results of the test match against the random network:
Score of lc_new vs lc_base: 24 - 1 - 8 [0.848] 33

[Akababa]

Nice results! Did you have a chance to play against the model yourself to see if it may be overfitting to "high-level" games? I believe I had a problem early on where all my training data had little material variance so the model had no chance to learn material imbalances (and the idea behind introducing variance/dirichlet noise to the self-play generator could partly be to "remind" the network what bad positions look like, improving training stability).

Another idea to eliminate the white/black bias that @Zeta36 brought up is to flip the board (isomorphic transform the space of "black-to-move" positions into "white-to-move" positions). I think this also introduces extra regularization at no cost.

[glinscott]

It's definitely still making a ton of mistakes. Here is an example game with the new network as white, random weights as black:

1. d4 e6 2. c4 d6 3. Nc3 {0.52s} Bd7 {0.51s} 4. e4 {0.54s} Qe7 5. Nf3 {0.50s}
Nh6 6. Be2 f6 7. O-O f5 8. exf5 g5 9. fxg6 Bb5 10. cxb5 Qf6 11. Re1 Qxg6 12. Bd3
Ke7 {0.71s} 13. Bf4 {0.50s} Kf6 14. a4 Be7 {0.50s} 15. a5 Rf8 16. h3 Qh5
17. Bc4 {0.53s} Qc5 18. dxc5 {0.52s} Kf5 19. Qd2 {0.52s} Kg6 20. Bxe6 {0.52s}
Kh5 21. cxd6 {0.50s} Bf6 22. Nd5 {0.51s} Bc3 {0.50s} 23. Qxc3 Rh8 24. Qe5+ Nf5
25. Bf7# {White mates} 1-0

You can see that after 12. Bd3 Ke7 the queen is hanging, but the network doesn't see it. I suspect setting the network to predict just the move SF played is hurting here. When training in self-play, it learns to predict the probabilities of all the moves visited by UCT, which seems much more robust. Still this seems like a solid baseline. Good proof the code is mostly working too :).

I've uploaded these weights to https://github.com/glinscott/lczero-weights/blob/master/best_supervised_5_64.txt.gz if others are interested.

The match score against the random network was:

Score of lc_new vs lc_base: 95 - 1 - 4  [0.970] 100
ELO difference: 604

(amusingly it lost the very last game)

[Zeta36]

Curiously the game seems to play more or less like @Akababa best results. Did your model already converge @glinscott? Maybe we could be facing some kind of limitation in the model we are using (following AZ0).

[glinscott]

@Zeta36 I don't think it's converged yet, but I've had to drop the training rate twice so far. So probably getting close:

[Zeta36]

I see. And don't you think it's a little strange that with so a very good convergence (in both policy and value head) the model still does so many (and clear) errors? @Akababa tried lot of times (also with very good convergence and with very much rich and big PGN files) but the model never was able to get rid of all blunders (and reach nearly no strategy in a long range). Maybe there could be something profound and related with the model itself that makes it unable to learn in a good manner (something DeepMind didn't explain in its last paper or something like that).

[Akababa]

I actually think it's very reasonable for the model to blunder when trained on one-hot Stockfish moves in very balanced positions. I'd say this is a limitation of supervised training in this fashion, and as @glinscott pointed out training on MCTS visit counts would be more robust because it's a good local policy improvement operator. But iirc leelazero reached a decent level by this same method so it could simply be lack of MCTS playouts or chess requires more layers.

Also without the validation split it's impossible to measure overfitting.

[gcp]

With only 80k games there is very likely to be a vast overfit in the value layer. You can control for this by lowering the MSE weighting in the total loss formula. Or by having more games (I'm still generating a ton), which is far better.

From this discussion I don't see why you think the network will converge to always returning 0.5 (or 0 in -1...1 range) though. It will reach that point quickly, but it will also be able to see that when one side has much more material (which is a trivial function of the inputs), the losses drop heavily when predicting a win for that side. And so on.

It is harder/slower to train to imbalanced categories but it certainly isn't impossible.

[gcp]

But iirc leelazero reached a decent level by this same method so it could simply be lack of MCTS playouts or chess requires more layers.

It also had a ton more data! About 2.5M games times 8 rotations (not possible to use rotations in chess).

[Zeta36]

@gcp, but once the model converge quickly (and deeply) into ignoring the input and saying always 0, the weights of the last FC layer will be so very small that any try to backpropagate any gradient would be almost negligible, isn't it?

"Go" does not have this (theoretical) problem since it has no zero result game (no draws).

About the number of games: @Akababa tried with a huge dataset of really big PGN files in here: https://github.com/Zeta36/chess-alpha-zero, and even he got a good convergence of mse and policy the model could play more or less "good" games although he was not able to remove all blunders nor to get a model able to play some strategic game in the long range.

What do you think?

[gcp]

but once the model converge quickly (and deeply) into ignoring the input and saying always 0, the weights of the last FC layer will be so very small that any try to backpropagate any gradient would be almost negligible, isn't it

But why would it converge that way? Not all games are drawn. The mispredictions on the won or lost games will still cause big gradients. The predictions of 0 on the drawn games will cause no gradient. Predicting 0.1 on a game that was drawn will produce a tiny gradient compared to mispredicting a win as a draw. Still plenty of room to make the distinctions, as there is strong pressure (and actual gradient direction) on the network to correctly predict the 40% of games that are not drawn.

"Go" does not have this (theoretical) problem since it has no zero result game (no draws).

Predicting 0 is still an easy way to get a quick loss on MSE error compared to always predicting a constant value, or predicting randomly. So I'm not sure this argument even works!

About the number of games: @Akababa tried with a huge dataset of really big PGN files in here

This page talks about "1000" games or "3000" games. You want 2 orders of magnitude more, or you will get MSE overfitting, as I already pointed out.

[Zeta36]

@Akababa tried with a lot more than 3000 games (even when in the readme he only talks about 1000 games).

About the quick convergence:

Imagine you have a worker playing self-games (or a parser of PGN files). You get a chunk of 15.000 movements for example. In average you will have more or less 5000 movements with z=-1, 5000 with z=0, and 5000 z=1.

Then you run the optimization worker an it reads the 15.000 movements to backprop. The loss function is mse so there is a clear an easy(fast) way to reach a local minimum when the model ignore the input (the board) and always says 0 (the mean) as result (simply by taking the weights of the last FC layer to values very near 0).

In this case, the optimization will reach quickly a high accuracy rate of 1/3 (33%) and a low mse loss of 0.66. After very few steps the model would get stuck with this function and no further improvement could make the model to take him out of this fast (and deep) convergence (because the weights of the last FC layer will be so very small that any try to backpropagate any gradient would be almost negligible).

I'm just talking theoretically and if the dataset has not bias but an almost perfect proportion between win, loss and draws (-1,0,1) and if we use a really big dataset unable to be overfitted.

[gcp]

because the weights of the last FC layer will be so very small that any try to backpropagate any gradient would be almost negligible).

I just don't see why this would happen for the reasons already stated. And unless there were bugs, does the practical result from @glinscott not show that it does not?

With 40% of the games producing a strong gradient towards anything that remotely correlates with the population count of many input planes[1], and all draw games producing an ever tinier gradient towards "always 0", how could you get stuck deeply in a local minimum? It just sounds so weird.

[1] This is why my dataset has no resignations, FWIW.

[Zeta36]

@glinscott results are based in a very biased dataset (with doubles z=1 respect to z=-1) and with a small an easy to overfit number of movements.

I'd be great to check this with a very much big number of movements and with an unbiased dataset (with near equals numbers of z=-1, 0 and 1).

[Akababa]

@Zeta36 What about using the flip-policy to regularize out bias? It's equivalent to x2 data augmentation (which is actually possible in chess @gcp) and possibly better. Although I'm not sure it's a huge concern to begun with as the model should be complex enough to recognize imbalanced positions so as long as you show it some black winning it should kill the bias in the long run. Training might be marginally faster with a maximal entropy training set though.

[gcp]

https://sjeng.org/dl/sftrain_clean.pgn.xz now has about 155k games.

[Error323]

Ok @glinscott I have interesting results here. I changed the tensorflow training code to use a separate training and validation set. Sampled 500K positions from @gcp 's 80K games, converted them to raw tensorflow protobuf format and split them into 75% train and 25% test. Getting to an accuracy of 70% now using the default network (64 filters, 5 residual blocks). I changed the learning rate to 0.005 at ~100K steps.
Also, I think augmenting the data as @Akababa suggested by board flipping is a good idea. We're already flipping the board to the current color. I'll push my code changes after some cleaning. Finally I'll run some games against random.

Sample output:

step 137400, policy loss=1.88179 mse=0.0328728 reg=0.593448 (6585.84 pos/s)
step 137500, policy loss=1.88024 mse=0.0328149 reg=0.593368 (6593.21 pos/s)
step 137600, policy loss=1.88207 mse=0.0329415 reg=0.593287 (6589.3 pos/s) 
step 137700, policy loss=1.88264 mse=0.0329211 reg=0.593206 (6608.15 pos/s)
step 137800, policy loss=1.87931 mse=0.0328492 reg=0.593126 (6591.49 pos/s)
step 137900, policy loss=1.87931 mse=0.0328397 reg=0.593046 (6598.33 pos/s)
step 138000, policy loss=1.88183 mse=0.0329394 reg=0.592966 (6594.76 pos/s)
step 138000, training accuracy=68.9551%, mse=0.105712
Model saved in file: /home/fhuizing/Workspace/leela-chess/training/tf/leelaz-model-138000
Leela weights saved to /home/fhuizing/Workspace/leela-chess/training/tf/leelaz-model-138000.txt 
step 138100, policy loss=1.88048 mse=0.0328839 reg=0.592886 (5102.31 pos/s)
step 138200, policy loss=1.87727 mse=0.0328062 reg=0.592806 (6536.92 pos/s)

[Zeta36]

Someone in reddit (https://www.reddit.com/r/MachineLearning/comments/7qdwb5/is_it_possible_to_train_the_value_output_in_a/) also commented about the problem of the value head tending to the mean (zero):

"I encountered a problem like this when I was developing my own chess implementation of AGZ (GitHub here:https://github.com/trebledawson/chess, but I'm not incredibly happy with how it turned out). My conclusion was that the size of my neural network was too small, and was thus underfitting the value output model. I attempted to increase the number of parameters until it was as large as my computer could store, but I still experienced the v-output tending toward the mean (i.e. 0)."

[gcp]

I changed the tensorflow training code to use a separate training and validation set.

I couldn't immediately tell from the code, but are you splitting on entire games? If you randomly partition positions, you'll need to make sure positions from training games aren't in the validation set.

(Else the network will just remember where the pieces ended up and what the game result was, and you won't detect that overfit. It'll easily do that at intermediate sizes.)

[Error323]

I samled 500K games from the ChunkParser in _parse.py_ see https://github.com/Error323/leela-chess/blob/master/training/tf/leela_to_proto.py. And used the first 75% for training and the last 25% for testing. There might be a bit of overlap, a game from each worker?

[gcp]

I guess it depends on the chunk sizes? The games aren't fed in in order: https://github.com/Error323/leela-chess/blob/f3646e28c6a544b519da8e8f9f7206f552354da1/training/tf/parse.py#L105

[glinscott]

Ok @glinscott I have interesting results here. I changed the tensorflow training code to use a separate training and validation set. Sampled 500K positions from @gcp 's 80K games, converted them to raw tensorflow protobuf format and split them into 75% train and 25% test. Getting to an accuracy of 70% now using the default network (64 filters, 5 residual blocks). I changed the learning rate to 0.005 at ~100K steps.

@Error323 wow, that is awesome! I guess I really limited myself with starting with a smaller subset of the games and keeping the weights from that run, the accuracy has still only made it up to 33%.

I added you on https://github.com/glinscott/lczero-weights, if you want to upload the weights there. Will be very interesting to see how it does against the version I trained. My hope is it destroys my weights :).

[glinscott]

I guess it depends on the chunk sizes?

@gcp the chunk sizes are 15000 positions per chunk for supervised learning currently: https://github.com/glinscott/leela-chess/blob/master/src/main.cpp#L308

[Error323]

Let's see if I understand this correctly. A chunk during supervised training contains 15K samples, I spawned 11 workers, each worker processes a full chunk before moving to the next random one. Each next(gen) call in leela_to_proto.py samples a position from one of the 11 chunks. So on average there would be 11*(70/2) positions of game overlap, given an average of 70 ply per game, which corresponds to 0.0031%. What do you think @gcp

[jkiliani]

Would it be a reasonable assumption that leela-chess with a good supervised network should be able to compete with Giraffe (2400 Elo)?

[gcp]

I guess another way of saying this is that each chunk contains ~215 games, so assuming they're drawn from equally (not actually sure that is guaranteed?), we expect the first 161 to end up in training, and the rest in test, and maybe a game per chunk split between the two?

Doesn't sound like it would be much of an issue then.

Note that the overfit & validation stuff is mostly there to make judging things during supervised learning easier (which is why it was never completed with proper splits etc).

[glinscott]

Btw, playing the network that trained overnight (and only made it to 33% accuracy) is currently winning handily against the previous best version: 16-3-9. That's a great sign that the network is being used well by the search.

One thing we might need to do is introduce a minimalist opening book to the testing matches though. Google was using many threads to feed the 4 TPUs, while we are using only a few with one or two gpus. That doesn't add much non-determinism into the games when played without noise. It's enough to avoid duplicate games, but they take a while to diverge from each other (eg. 10-11 moves into the game). The cool thing is it has learned the Gruenfeld and Queen's gambit declined from raw SF games. That speaks well to SF I guess :).

Edit, the new network won 49-13-38, so definitely improving!

[gcp]

@glinscott That shouldn't be needed. The current code is missing a call, which is why it does not diverge well: https://github.com/glinscott/leela-chess/blob/9b1e319eda079b66bf9b742ed5005877ade1e541/src/UCTSearch.cpp#L133

Which should be calling randomize_first_proportionally:
https://github.com/gcp/leela-zero/blob/07b47a2e74febb0d6e2b10597690c19404cdc48d/src/UCTSearch.cpp#L149

Note that in AZ this wasn't move number dependent, they left it always on for self-play and always off for testing matches. I think you want something like an UCI option to toggle this.

If you introduce an opening book you risk not finding out if 1.d4 is better than 1.e4 or the French better than the Caro-Kann, etc :-)

[glinscott]

@gcp Oh, interesting! I thought they only used the Dirichlet noise in AZ. That's my mistake, I will re-add that in. That probably hurt the training game generation a fair bit as well...

[gcp]

For reference, relevant section from the paper:
"During training, each MCTS used 800 simulations. .... Moves are selected in proportion to the root visit count."

Earlier, there is "The search returns a vector π representing a probability distribution over moves, either proportionally or greedily with respect to the visit counts at the root state"

They don't say what the "greedily" mode is for, but logically it's what they use for evaluation games against other opponents. Note that with no Dirichlet noise and no proportional choice, the program is almost deterministic. Which means you can't play testing games between two networks, as they will repeat the same game. And guess what, one of the differences between the AlphaGo Zero and Alpha Zero is that the latter no longer plays testing games to promote a network...

[jkiliani]

You could probably raise playing strength quite a bit over the training setting (temperature=1) without making the program deterministic, with a temperature setting around 0.25 to 0.5. This would effectively suppress moves with very small visit counts from selection, while still giving moves with visit counts close to the PV a reasonable chance of being picked.

[Error323]

Final result from training 93% I lowered the learningrate 3 times: 0.05, 0.005, 0.0005, 0.00005. ~Now I wanted to run some games with cutechess-cli, but the current master crashes for me @glinscott :( not sure why yet, was tinkering over ssh~ I'm a dummy, it works. I'll upload the weights.
`

[gcp]

Accuracy of 93% sounds pretty crazy. It's actually predicting ~15 ply StockFish searches with 93% accuracy? How damn strong is this already?

[Error323]

I don't know o_O I can't run lczero atm. But yeah it really does sound crazy, almost think I messed up somehow :sweat: @glinscott could you run a tournament please? I don't have time for the next 8 hrs.

@gcp are the games from your dataset diverse enough I wonder?

[gcp]

@gcp are the games from your dataset diverse enough I wonder?

What does "diverse" mean? It's using a book to force diverse openings but you won't get totally crazy 5 pieces hanging positions of course, as it's Stockfish vs Stockfish. Using weaker players would weaken the policy part of the network.

[Error323]

Yeah I ment in diversity of it's openings. OK then it's good. I guess it'll be decent against minimax based engines, but contains huge gaps in its gametree knowledge?

[Error323]

Against random:

Score of supervised vs random: 66 - 6 - 28 [0.800] 100
Elo difference: 240.82 +/- 65.32
Finished match

[Zeta36]

Accuracy of 93% with a dataset of 500k movements and even so it losses 28 times against a random network? It sounds a little bit strange, sin't it? In here https://github.com/Zeta36 the random network was 100% defeated much (much) more quickly.

[Error323]

Yeah I agree, it's strange. I'll investigate. @Zeta36 I don't see anything wrong, I parsed the datasets and they seem fine. I guess it just has too many gaps in its knowledge that leave it very vulnerable. Or maybe there's a problem with the UCTSearch? I'm gonna increase the dataset.

[glinscott]

@Error323 I'm playing a match of your 93% network against the network I trained that had reached 33% accuracy from above (uploaded to the lczero-weights repo as best_supervised_5_64.txt.gz), and currently the 33% accuracy is winning handily (11-0-5). Also, I haven't tried the 33% network against the random network, but the previous iteration of this network (at about 20% training accuracy) won 95-1-4 against random network.

I'm not sure what is going wrong though. Perhaps overfitting? But that seems unlikely if the validation split is working.

[Error323]

Mja it felt to good to be true. I think the dataset I generated contains 500K positions from full games. The sampling method that generated the binary proto data is bad. There are approx 70*80K=5.6M positions in the set of which I sampled only the first 500K with all positions from those games.

We should split the *.gz chunks files into _p_, (1-_p_) and then sample _pN_ positions for training and _(1-p)N_ for testing in a truely random fashion.

[Error323]

Oke, I messed up :( The reason the accuracy kept going up is because the average accuracy was calculated incorrectly:
https://github.com/glinscott/leela-chess/blob/master/training/tf/tfprocess.py#L196-L204

[glinscott]

@Error323 ah, good catch. But that's just a constant multiplier, it shouldn't cause a runaway type of effect.

Also, I trained another network up from scratch on the 150k games from gcp, it reached better accuracy than the previous network (they were both measured with the wrong code), but it plays much weaker than the previous best. So something interesting going on here, overfitting is my suspicion.

Here were the results:

step 36000, policy loss=2.77798 mse=0.055978 reg=0.369072 (2820.14 pos/s)
step 36000, training accuracy=41.2207%, mse=0.277543

And then against the current best:

Score of lc_new vs lc_base: 10 - 57 - 33  [0.265] 100
ELO difference: -177

[gcp]

step 36000, policy loss=2.77798 mse=0.055978 reg=0.369072 (2820.14 pos/s)
step 36000, training accuracy=41.2207%, mse=0.277543

Note the training MSE=0.05 vs the test MSE=0.27. So this is a total overfit.

https://github.com/glinscott/leela-chess/blob/master/training/tf/tfprocess.py#L84

Try lowering the factor for the mse_loss by a factor 10 there, or increase the regularizer above 10-fold.

[Error323]

I think the runaway was caused by consecutive positions, which were captured perfectly by the input history planes...

Yeah the MSE is important to get right as it guides the MCTS.

[kiudee]

To expand on what @gcp said for AlphaGo Zero the DeepMind team wrote:

Parameters were optimized by stochastic gradient descent with momentum and learning rate annealing, using the same loss as in equation (1), but weighting the MSE component by a
factor of 0.01.

So it appears they determined the effect on the total loss was too high. In the AlphaZero paper they also wrote they reused many of the parameters for chess.

[glinscott]

Thanks @gcp and @kiudee. I'm trying with mse at 0.1 weight now.

[glinscott]

Well, 0.1 weight still seems to be overfitting a bit. But it seems better now. Still, the newly trained networks are losing to my original one.

step 169900, policy loss=2.77919 mse=0.0678728 reg=0.33725 (3426.75 pos/s)
step 170000, policy loss=2.78106 mse=0.0682819 reg=0.337243 (2552.92 pos/s)
step 170000, training accuracy=27.3958%, mse=0.0758204

Match vs best:

Score of lc_new vs lc_base: 7 - 24 - 25  [0.348] 56

[Zeta36]

If the trained network loses against the random one there has to be a profound problem with the model (or with the MCTS implementation). I don't think this issue is due to over-fitting.

Also I think the mse error is scaled down by a factor of 4 so the real one would be: 4*0.08 ~ 0.33, isn't it?

[glinscott]

@Zeta36 it's not losing against random, it defeats it nearly 100%. It's losing against the previous best network I did supervised training with (but on about half the games).

[Zeta36]

I'm sorry, I misunderstood what you said.

[Error323]

@glinscott which weights are you using as reference? Then I can try some tests as well and we can compare them properly. https://github.com/glinscott/lczero-weights/blob/master/best_supervised_5_64.txt.gz those?

[kiudee]

I started experimenting with the training of the network and noticed that it is very slow (~4000 steps per hour) on my machine (6 core i7 and GTX 1080). It appears the GPU is almost idle for most of the time, while the worker threads use 100% of my CPUs.
@glinscott which machine(s) are you using for training and what wallclock time did you need for the run posted before?

To prevent questions whether tensorflow is using the GPU:

2018-01-18 09:54:25.559595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.8855
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 6.96GiB
2018-01-18 09:54:25.559626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Test parse passes
2018-01-18 09:54:26.139863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)

[glinscott]

@glinscott which weights are you using as reference? Then I can try some tests as well and we can compare them properly. https://github.com/glinscott/lczero-weights/blob/master/best_supervised_5_64.txt.gz those?

Yup, it's the ones I uploaded as best_supervised_5_64.txt.gz.

[glinscott]

@glinscott which machine(s) are you using for training and what wallclock time did you need for the run posted before?

@kiudee I had increased the default batch size by a lot to 2048, but I think this may have been a mistake. You might want to try reducing it to 512 or 256. I'm training on a Titan X currently, and it took about a day to train up to 170k steps at 512 batch size.

[glinscott]

I've added the --randomize option in now, and played training games last night with the current best network. I'm going to try training a new network based on those training games, and see how we do :).

[Error323]

Ok I'm creating a massive dataset from KingBase2017 of 1.5M games with 500K black wins, 500K white wins and 500K draws. I used pgn-extract as suggested by @gcp which is excellent. I'm beginning to think unbalanced datasets are indeed a problem for learning. We may need to remember that when generating games from selfplay.

pgn-extract pgn/mega.pgn -Tr1-0 --nobadresults -D -7 --stopafter 500000 -# 5000 -s
pgn-extract pgn/mega.pgn -Tr0-1 --nobadresults -D -7 --stopafter 500000 -# 5000 -s
pgn-extract pgn/mega.pgn -Tr1/2-1/2 --nobadresults -D -7 --stopafter 500000 -# 5000 -s

Now generating the chunks.

[gcp]

I don't really buy that the unbalanced sets are hard for learning (I mean, they make it harder, for sure, but this should still be trivial stuff for neural nets). But having 1.5M games will improve the overfit problem, though your source data will be a bit weaker.

FWIW the https://sjeng.org/dl/sftrain_clean.pgn.xz dataset is updated to about ~225k games now.

[Error323]

Well I agree with enough diverse data. But the current set is still quite small given the game complexity. The minimax game tree is sparse given stockfish vs stockfish and a significant portion is drawing. Also in the 150K set, about 10K is duplicate according to pgn-extract.

[gcp]

Also in the 150K set, about 10K is duplicate according to pgn-extract.

Ah, good to know, I'll switch the book to a uniform probability one.

[Error323]

Maybe it would also be interesting to randomize the depth of the stockfish opponent? Though not using a uniform distribution.

[gcp]

Maybe it would also be interesting to randomize the depth of the stockfish opponent?

It's the latest Stockfish playing itself at a fast timecontrol, the idea was to get a ton of the highest quality games possible (in a reasonable amount of time). I'm assuming 10+0.1 latest Stockfish on fast hardware is better than 2500 GM's at this point. Using a uniform book doesn't really help the variation, so I made the book a bit bigger, but it may be necessary to strip dupes afterwards anyway.

[Zeta36]

People, I don't want to bother you again with the same thing, but I'm still working in a way to train a supervised value head and as I said it's impossible avoid the model to deep into the easy function of ignoring the output and saying always 0 (the mean with 33% of accuracy):

Epoch 1/1
256/155602 [..............................] - ETA: 16732s - loss: 1.0572 - acc: 0.3359
512/155602 [..............................] - ETA: 16128s - loss: 0.9843 - acc: 0.3555
768/155602 [..............................] - ETA: 17647s - loss: 0.9229 - acc: 0.3646
1024/155602 [..............................] - ETA: 17876s - loss: 0.8946 - acc: 0.3633
1280/155602 [..............................] - ETA: 17758s - loss: 0.8846 - acc: 0.3539
1536/155602 [..............................] - ETA: 17762s - loss: 0.8763 - acc: 0.3496
1792/155602 [..............................] - ETA: 17598s - loss: 0.8758 - acc: 0.3410
2048/155602 [..............................] - ETA: 17480s - loss: 0.8709 - acc: 0.3394
...
9216/155602 [>.............................] - ETA: 16232s - loss: 0.8413 - acc: 0.3332
9472/155602 [>.............................] - ETA: 16230s - loss: 0.8406 - acc: 0.3336

The only way in which I can avoid this result is over-fitting the network (i.e.; memorizing the games results). But as soon as the movements grow the network is unable to learn anything (and being not able to memorize anymore it goes to the statistic 1/3 solution and stay there).

This last test I show you it's the same of https://github.com/Zeta36 but without the policy head. I mean, the model it's the same (l2 regulator, residual blocks, etc.) but I removed the policy head. The loss shown above is the mse (without scaling by 4 like you do here).

The model converge deeply after a little steps and get stuck in 33% accuracy (because 1/3 of times y=z=0, and the mean of -1,0,1 is also 0).

Maybe your fault in this tests here it's because the head is not able to be trained in a supervised way in a domain of 3 integer values. I know that I removed the policy head and that could be the cause of this fast (and logic) convergence but who knows.

[Error323]

Hey @Zeta36,

Could you be more specific about the size of the dataset? The exact location of the model used, what the inputs are? Is it this one? https://github.com/Zeta36/chess-alpha-zero/blob/master/src/chess_zero/agent/model_chess.py#L62 if so, where do those 18 planes come from? It seems like you don't store history.

@gcp I had the same idea indeed. We're gonna have to experiment, having the dataset is definitely good! My worry is that the gametree will contain gaps because stockfish vs stockfish contains a single evaluation function. It might be interesting then to select different engine opponents?

[kiudee]

I am currently training the network on the latest stockfish data with mse weight 0.01. Before chunking the games, I filtered all the dupes using pgn-extract.
This is the current state of the training:

A quick match at 40k steps against the best_supervised_5_64 network resulted in

Score of lc_new vs lc_base: 0 - 34 - 10

So, it’s not yet able to win against it. I will post an update as soon as I have new results.

[Error323]

So I sampled 1.5M uniqe, correct games from KingBase (1/3 white, 1/3 black, 1/3 draw) from which I generated 7.71M positions by randomly selecting every 16th or 17th pos. The trainingset (75%) and testset (25%) have guaranteed 0 games in common and amount to ~110GiB in raw tensorflow binary format. Final accuracy 40.33%.

Each dip in the graph represents a reduced learning rate by a factor of 0.1 and MSE loss weight factor was 0.01. It's currently beating best_supervised_5_64 with:

Score of lc_new vs lc_base: 67 - 14 - 19  [0.765] 100

I'm observing that each minor improvement in accuracy on the NN has huge perf gains against the baseline weights. @glinscott would you review PR #36 now as it is verified correct. Weights can be found here.

[Zeta36]

@Error323 the mse loss in the plot is scaled down by a factor of 4? If that's the case, then the real mse would be 0.17*4 ~ 0.68, isn't it? It's curious that if the NN (the head value) learns to say always near 0, we'd get an instant loss of (mse): 100/150 ~ 0.66 and an accuracy of ~33% (1/3).

[Error323]

@Zeta36 mse is indeed scaled by a factor of 4. We do observe an accuracy that's higher though (it's quite challenging playing against the network now). I'm generating a pure black/white win dataset which I'll train using the same method to see how that would perform.

[glinscott]

I'm observing that each minor improvement in accuracy on the NN has huge perf gains against the baseline weights. @glinscott would you review PR #36 now as it is verified correct. Weights can be found here.

@Error323 congratulations! That's great progress.

[kiudee]

I have finished my training run on the Stockfish games over the weekend and here are the "final" training curves:

I took the network with the highest accuracy (888k steps) and played it against best_supervised_5_64:

Rank Name      Elo    +    - games score oppo. draws 
   1 lc_new     77   30   29   100   74%   -77   35% 
   2 lc_base   -77   29   30   100   27%    77   35% 

My parameters were:

• 0.01 * self.mse_loss
• BATCH_SIZE = 512
• learning_rate = [0.05, 0.01, 0.005, 0.001]
• I removed all game duplicates but did not try to balance the dataset.

If you want to play around with that network yourself, here are the weights:
best_stockfish_5_64.zip

[Error323]

Tried a deeper network with 128 filters and 10 residual blocks. It only got a few percent better, but those few percent make a huge difference:

Also, I trained 2 at the same time and I swear my system was about to lift off into space. Results against baseline:

Score of lc_new vs lc_base: 79 - 8 - 13  [0.855] 100

I've got many more results, trying to see what would be a good tradeoff when learning our first network tabula rasa. Where shall I post those results?

[kiudee]

I think it all comes down to the rate at which we are able to produce self-play games. What is your approximate ms/move for a 10x128 network?

[Zeta36]

Have anybody checked how the model plays after that huge SL training? I mean, DeepMind always report (in Go at least) that after a supervised learning their model was a really good player.

What I want to say is that (in my opinion) it is not so important the loss results (nor the game against a baseline) but the real ELO a huge SL is able to reach with the model you are working on.

I think before going to self-play training we should check the model learn to play chess in a SL manner. In this sense, if the model you are working on is really good (as loss says) after a training of more than 500k movements it should be able to not blunder and to play a really good level game.

Can anybody show over here if the trained model plays well against you (using any Chess GUI app as Arena for example: http://www.playwitharena.com/)?

I comment this because we also got really good loss results in a SL way in https://github.com/Zeta36/chess-alpha-zero but then the model finally never was able to get rid of all blunder movements.

Maybe you are happy with the loss results and then the model is unable to play well even after a 500k training process. If that's the case I don't think a million of self-play games can do much better.

In summary: as DeepMind pointed out a correct model should be able to reach a really impresive ELO just with a SL training process. I do not recommend any self-training in the meantime.

[Error323]

What is your approximate ms/move for a 10x128 network?

@kiudee It was equal to the 64x5 network, except that my GPU went to 96% utilization. My syste can handle 7200 forward/backward passes per second per GPU.

In summary: as DeepMind pointed out a correct model should be able to reach a really impresive ELO just with a SL training process. I do not recommend any self-training in the meantime.

@Zeta36 I fully agree. I'm still kind of dissapointed with the performance of only 43%. I think we should stick with 64x5 and determine it's real ELO before continuing.

[lp--]

@Zeta36 Without randomization with fixed number of playouts it is producing single game when playing with itself. Level is extremely low. Seems it picked up pattern of how to put pieces and pawns on board, but don't understand value of pieces, how to mate, etc

[Zeta36]

It's very complicated to understand how it's possible that a model trained after 500k with a so good convergence is not able to understand the value of the pieces. Something has to be wrong.

[Error323]

@Zeta36 Have you tried playing the latest version yourself? it's quite fun.

[gcp]

More specifically, if you play a few moves and let it go up a piece (or down a piece), does the evaluation move away from 0? 2 pieces? etc?

What if you give it a large development advantage?

[gcp]

I updated my dataset, there's now about 320K de-duped games. It's hard to conclude from the above, but is this giving better or worse results than human games? I guess to there still being more human data (1.5M games) that is also a bit more varied, that might help a lot?

[Zeta36]

@Error323 I don't have the environment available right now. Could you please paste over here an animated gif with some games? You can use this online tool: http://www.apronus.com/chess/wbeditor.php

Once you replicate the movements of the game just click "save" and then "animate diagram". Finally you can upload the gif file in a comment in here.

[Zeta36]

This, for example, was our best achievement (in a SL manner) in https://github.com/Zeta36/chess-alpha-zero

In this game @Akababa (black, ~2000 elo) played against the model (white):

By the way, the weights of this model are uploaded in our repo.

[jkiliani]

You may also take a look at this: https://github.com/gcp/leela-zero/issues/696 for a possible strength boost. In general, it looks like good search parameter tuning may make a big difference.

[lp--]

This is the game it plays with @kiudee weights
./lczero -t 1 -w leelaz-model-888000.txt --start train --noponder -p1600

1. d4 Nf6 2. c4 e6 3. Nc3 Bb4 4. Qc2 O-O 5. a3 Bxc3+ 6. Qxc3 b6 7. Bg5 Bb7
8. e3 d6 9. Ne2 Nbd7 10. Qd3 h6 11. Bh4 Ne4 12. Be7 Qxe7 13. f3 Ng5 14. h4
Nxf3+ 15. gxf3 c5 16. Rg1 Kh8 17. Rxg7 Kxg7 18. d5 Kh8 19. Qc3+ e5 20. Ng3
Qxh4 21. Bd3 Rg8 22. Kf2 Rxg3 23. Rg1 Rg5+ 24. Kf1 Rxg1+ 25. Kxg1 Rg8+ 26.
Kf1 Qh2 27. Ke1 Rg1+ 28. Bf1 Kg8 29. Qd3 Nf6 30. Qf5 Qg2 31. Qd3 Qh2 32.
Qf5 Qg2 33. Qd3 Qxb2 34. Qe2 Ng4 35. fxg4 Qxa3 36. Kf2 Qb3 37. Kxg1 Qb1 38.
Qd3 Qe1 39. Kg2 e4 40. Qxe4 Qd2+ 41. Kf3 Qd1+ 42. Kf2 Qd2+ 43. Kf3 Qc1 44.
Qe8+ Kg7 45. Be2 Bxd5+ 46. cxd5 c4 47. Kf4 c3 48. g5 Qg1 49. gxh6+ Kxh6 50.
Bb5 Kg6 51. Qg8+ Kh6 52. Qxg1 c2 53. Qg5+ Kh7 54. Bd3+ Kh8 55. Bxc2 a6 56.
Qe7 Kg7 57. Qxd6 a5 58. Qb8 b5 59. d6 Kf6 60. Qxb5 Ke6 61. Qc6 Kf6 62. Qb5
Ke6 63. Qc6 Kf6 64. Qc7 Kg7 65. d7 a4 66. Bxa4 Kf6 67. Qb6+ Kg7 68. Qc7 Kf6
69. Qc8 Kg7 70. Kf5 f6 71. d8=Q Kf7 72. Qcc7+

[kiudee]

@gcp Using my weights trained on Stockfish games, it seems to have no concept of material. This is what happens if you gift it a knight in the beginning. The eval does not change and still favors white. To make matters worse, through searching more deeply it starts to favor a worse move.

position startpos moves e2e4 e7e5 g1f3 b8c6 f3e5
go
NN eval=0.452850
Playouts: 4618, Win: 43.42%, PV: c6e5 d2d4 e5c6 b1c3 g8f6 f1d3 d7d6 e1g1 f8e7 d4d5 c6e5 d3e2 e8g8 f2f4 e5g6
Playouts: 9187, Win: 43.64%, PV: c6e5 d2d4 e5c6 b1c3 g8f6 f1d3 d7d6 e1g1 f8e7 d4d5 c6e5 d3e2 e8g8 f2f4 e5d7 g1h1
Playouts: 13780, Win: 43.79%, PV: f8c5 e5f3 g8f6 d2d3 d7d6 b1c3 a7a6 f1e2 e8g8 e1g1 f8e8
Playouts: 18371, Win: 43.86%, PV: f8c5 e5f3 g8f6 d2d3 d7d6 b1c3 a7a6 f1e2 b7b5 e1g1 e8g8 a2a3 f8e8

@Error323 What happens if you do a similar test on your weights trained on human games?

[Zeta36]

I'm pretty sure the issue comes from the head value.

I've been thinking about training the value output in a supervised way no just with the result game z=[-1,0,1], but with a real score (calculated by Stcokfish). In Python it's very easy to create this dataset, it just involves to use the "python-chess" library to connect to Stockfish and get the evaluation of any board state (in any game inside the PGN files):

handler = chess.uci.InfoHandler()
engine = chess.uci.popen_engine("C:\\xxxxxxx\\stockfish-8-win\\Windows\\stockfish_8_x64.exe")
engine.info_handlers.append(handler)
engine.position(node.board())
engine.go(movetime=1000)
z = handler.info["score"][1].cp / 100.0

Unfortunately I don't have GPU right now to train.

I guess you could do this same thing in C++ easily and check for the model to be able to predict the real value score of a board state. If the head value is not able to converge with this really supervised value dataset, then we would have a problem with the model used (and DeepMind would have probably some secret no commented in their last paper).

[gcp]

I've been thinking about training the value output in a supervised way no just with the result game z=[-1,0,1], but with a real score (calculated by Stcokfish).

People who tried this in Go always got considerably worse results. Even AZ doesn't predict the UCT search outcome value, but the eventual game value, so it seems to apply to chess as well.

[Zeta36]

But the model value head is precisely the one in charge saying how good a state board is, so I don't know why it should not converge using a real score calculated by Stockfish (??) Do you have some theoretical explanation for what you said (a part of that practical experiments you comment)?

The eventual game result has a complete correlation with the current board state value, isn't it?

Anyway I think it's an easy test you can do in order to figure out if the model is able to learn at least the piece values.

[jkiliani]

Piece values are a human crutch to an evaluation function in chess. A neural network based engine is going to learn to evaluate position purely on whether they statistically lead to a win, draw or loss. Of course material has a significant part in this, but the neural network will eventually be able to evaluate the relative importance of material and positional factors much better than Alpha-Beta engines. Using handcrafted evaluation functions instead of simple win-loss seems unlikely to improve a supervised network, since they introduce biases from the engine programmers into the network.

[Zeta36]

Well, piece values is a fact that the model will have to learn sooner or later. Question is just to check if the model is able to figure out this fact using a score so we can press and bias the value head in this direction.

Because if we try to train the model (in a good supervised manner) offering as input a very good approximation of the real state value, and even so the model is not able to learn well, then we'd know the model will not ever be able to learn a good function value.

It'd be just a fast checking of the model capacities.

[Error323]

@Zeta36, as @jkiliani said, stockfish's score is not a real score. The only real score is the one determined at the end of a game {-1,0,1}. And only through sampling of the statespace can it be learned. The AlphaZero algorithm is just an efficient way of doing so.

[Zeta36]

I know it's not a real score (or a definitive one), but it's an excellent evaluation function. If our model is going to be able to reach a good value function in anytime, it has to be able to imitate the Stockfish function value by a SL way as I propose. It's just a check of the capacities of our model.

[gcp]

so I don't know why it should not converge using a real score calculated by Stockfish

I'm not saying it won't converge, I'm saying the results (playing strength) tend to be worse.

[gcp]

Using my weights trained on Stockfish games, it seems to have no concept of material. This is what happens if you gift it a knight in the beginning. The eval does not change and still favors white. To make matters worse, through searching more deeply it starts to favor a worse move.

Interesting! Now, what happens if you find a game with some (say rook) endgame where one side is one or two pawns or a passer up?

In the pure SF training data, nobody will ever be a piece down in the opening unless it was a sac with compensation. This situation just doesn't exist as far as the network is concerned. That's why I'm interested in the other example, which would be something more likely to be known to the network.

In (regular) Leela, if you give a player handicap stones, the network thinks he is worse off than without. That's because if you get handicap stones in human games, you're a weaker player that is more likely to lose.

[Zeta36]

I'm not saying it won't converge, I'm saying the results (playing strength) tend to be worse.

Sure, @gcp. But we could use even so the Stockfish score to check if our model is able to learn good value functions. If the model converges with this score as you say, it'd show the model is strong enough, in the other case it'd signify the model will never be able to learn the chess strategies and facts (no matter how many game results you give to it).

Until now and after training with more than 500k movements the model has learn no strategy or concept of chess. That's strange and we can not just suppose more games will fix the problem.

[kiudee]

@gcp I looked for a rook endgame (in the dataset) which would be instructive and found the following one:
8/4R2p/6pk/5p2/8/4P1KP/1r3PP1/8 w - - 0 48

Here the networks thinks white is ahead, but allows the black rook to eat its pawns (it’s even its PV)

position fen 8/4R2p/6pk/5p2/8/4P1KP/1r3PP1/8 w - - 0 48
go
NN eval=0.424949
Playouts: 4130, Win: 68.41%, PV: g3f4 b2f2 f4e5 f2g2 e5f6 g2h2 e7g7 h2a2 g7e7 a2a1
Playouts: 8279, Win: 67.20%, PV: g3f4 b2f2 f4e5 f2g2 e5d6 g2d2 d6c6 d2c2 c6d6 c2d2 d6c6
Playouts: 12443, Win: 66.35%, PV: g3f4 b2f2 f4e5 f2g2 e5d6 g2d2 d6c6 d2c2 c6d6 c2h2 d6e6
Playouts: 16594, Win: 65.86%, PV: g3f4 b2f2 f4e5 f2g2 e5d6 g2d2 d6c6 d2c2 c6d6 c2d2 d6c6 d2c2
Playouts: 20683, Win: 65.53%, PV: g3f4 b2f2 f4e5 f2g2 e5d6 g2d2 d6c6 d2c2 c6d6 c2d2 d6c6 d2c2
Playouts: 24835, Win: 65.25%, PV: g3f4 b2f2 f4e5 f2g2 e5d6 g2d2 d6c6 d2c2 c6d6 c2d2 d6c6 d2c2
Playouts: 28899, Win: 64.84%, PV: g3f4 b2f2 f4e5 f2g2 e5d6 g2d2 d6c5 d2c2 c5d6 c2d2 d6c5 d2c2
Playouts: 33066, Win: 64.46%, PV: g3f4 b2f2 f4e5 f2g2 e5d6 g2d2 d6c6 h6g5 e7f7 d2c2 c6d6 c2d2 d6e5 d2e2
Playouts: 37090, Win: 64.29%, PV: g3f4 b2f2 f4e5 f2g2 e5f6 g2e2 e7e5 e2f2 e5e7 f2e2 e7e5
Playouts: 41278, Win: 64.14%, PV: g3f4 b2f2 f4e5 f2g2 e5f6 g2e2 e7e5 e2f2 e5e7 f2e2 e7e5
Playouts: 45336, Win: 63.95%, PV: g3f4 b2f2 f4e5 f2g2 e5f6 g2e2 e7e5 e2f2 e5e7 f2e2 e7e5
Playouts: 49441, Win: 63.76%, PV: g3f4 b2f2 f4e5 f2g2 e5f6 g2e2 e7e5 e2f2 e5e7 f2e2 e7e5
...
Playouts: 396917, Win: 61.23%, PV: g3f4 b2f2 f4e5 f2g2 e5f6 g2e2 e7e5 h6h5 e5e7 h5h6 e7e5
400001 visits, 5996806 nodes, 400001 playouts, 1412 n/s

bestmove g3f4

[Error323]

From the 64x5 kingbase 1.5M trained network using 40K playouts:

position fen 8/4R2p/6pk/5p2/8/4P1KP/1r3PP1/8 w - - 0 48
go
Playouts: 7351, Win: 54.63%, PV: g3f4 b2f2 f4e5 h6h5 g2g4 f5g4 h3g4 h5g4 e7e6 g4g3 e6f6 f2e2 e5d6
Playouts: 14996, Win: 53.90%, PV: g3f4 b2f2 f4e5 h6h5 g2g4 f5g4 h3g4 h5g4 e7e6 g4g3 e6f6 f2e2 e5d6
Playouts: 22675, Win: 53.76%, PV: g3f4 b2f2 f4e5 h6h5 g2g4 f5g4 h3g4 h5g4 e7e6 g4g3 e6d6 f2e2 e5f6
Playouts: 30047, Win: 53.94%, PV: h3h4 g6g5 e7d7 b2e2 d7e7 e2b2 h4g5
Playouts: 37402, Win: 54.14%, PV: h3h4 g6g5 e7d7 b2e2 d7e7 e2b2 h4g5
eval=0.408687, 40001 visits, 601109 nodes, 40001 playouts, 2987 n/s
bestmove h3h4

Edit: Oh 400K playouts:

Playouts: 399406, Win: 54.80%, PV: h3h4 g6g5 e7e6 h6h5 h4g5 h5g5 f2f4 g5h5 e6f6 b2e2 f6f5 h5g6 f5e5 h7h5 g3f3 e2c2 f3e4 h5h4 e5e6 g6h5 e4f5 c2c3 e3e4 c3c2 e4e5 c2c3 e6f6 c3c2 e5e6
eval=0.408687, 400001 visits, 6073605 nodes, 400001 playouts, 2984 n/s
bestmove h3h4

[gcp]

Here the networks thinks white is ahead,

Thus proving there is not simply deep convergence to say it's always a draw.

but allows the black rook to eat its pawns

Oops.

[glinscott]

@Error323 exciting that the bigger network is not slower, and performs better! If you want to post a lot more results, perhaps the forum (https://groups.google.com/forum/#!forum/lczero) is a good place?

[glinscott]

@kiudee very interesting results! Have you tried your network head-to-head vs @Error323's network trained on the human games?

What I've been running in the background is randomized self-play games from the best_supervised_5_64 network, and then training networks from those results. Unfortunately, with a small number of games (only 10,000 so far), it is far too few games. The network ends up memorizing everything it appears.

[kiudee]

@glinscott Running it right now. The first 2 games were won by the Stockfish network.

[Error323]

What really bothers me is that the 64x5 network has about 700K weights. We're training on sparse images of 8x8 pixels and 120 channels. I feel it should predict the human moves much better from the 1.5M games. I bet many weights are just useless.

edit: Perhaps we should experiment some with squeezenets: https://arxiv.org/pdf/1602.07360.pdf or other different architectures.

[Error323]

@Zeta36 these results were learned from the KB dataset Black/White and Black/White/Draw. So the draw games pull down the MSE, but don't seem to affect the accuracy much.

[Zeta36]

Mmmmm...I don't know if I'm wrong but accuracy is related only with the policy head, isn't it? I think it is normal accuracy gets no affected by the dataset in this sense. But the pull down in MSE it's a real proeblm, moreover because I think you have to multiply it by a factor of 4 (~0.8 & 0.65).

correct_prediction = tf.equal(tf.argmax(self.y_conv, 1), tf.argmax(self.y_, 1))
correct_prediction = tf.cast(correct_prediction, tf.float32)
self.accuracy = tf.reduce_mean(correct_prediction)

By the way, if you look to the beginning of this issue you will see I commented:

In fact I did a toy experiment to confirm this. As I mention the NN was unable to improve after reaching 33% accuracy (~0.65 loss in mean square). And this has sense if the NN is always returning 0 (very near zero values). Imagine we introduce a dataset of 150 games: ~50 are -1, ~50 are 0 and ~50 are 1. If the NN learns to say always near 0, we get an instant loss of (mse): 100/150 ~ 0.66

It's interesting compare your results of 0.65 (after scaling up by 4) in MSE and what I say.

[Error323]

Well it should get affected when backpropagating right? Both heads move backwards through the residual blocks. The dual head both saves compute time and both heads regularize eachother. At least that's how I understand it.

[Zeta36]

But if the last FC layer of the value head converges quickly enough into very low weights no backprop gradient will be big enough to affect the rest of the model (only the policy part will contribute after that moment).

I'm not sure if I'm right or not, but we have now (after several training tests) some empirical facts:
1) The model is not able to learn any chess concept or strategy even after convergence and with huge amounts of data used.
2) The MSE normally converges very fast and near of the 0.65 (after scaling up by 4). You just have to look into your own plots, @Error323.

So, maybe there is some kind of problem with the model and in this case more training data will not resolve the issue (many people is supposing just going like crazy into self-play will fix this).

[kiudee]

Here is the result from the 100 game match between the human and the
Stockfisch network:

Rank Name Elo + - games score oppo. draws
1 stockfish 14 27 27 100 55% -14 49%
2 human -14 27 27 100 46% 14 49%

Seems like they are of similar strength. I can post some games when I am on
my PC again.

[kiudee]

Here is a typical game between the human network (white) and the Stockfish network:

Here is one where human (white) wins against Stockfish in the Spanish opening:

The rest of the games you can find here (lc_new is Stockfish and lc_base is human).

[gcp]

The MSE normally converges very fast and near of the 0.65 (after scaling up by 4). You just have to look into your own plots, @Error323.

You keep repeating this but:

a) The results after normalizing to have equal amounts of -1, 0 and 1 results are similar.
b) There are clearly positions where the network can have learned who is ahead and it is indeed not returning 0 values for those.
c) Doing SL with lowered MSE factor helps the policy net to develop structure in the residual stack before the value net does.

[gcp]

Here is a typical game between the human network (white) and the Stockfish network

Maybe this is my human tendency to see patterns where there are none, but it does seem to be defending its pieces (most of the time)? It ends up hunting the queen a few moves, and she moves out of the way.

[kiudee]

It does defend its pieces, but it seems more because it prefers patterns where this is the case, rather than by material. But this is just a guess from watching some of the games.

[Zeta36]

That game is probably result only of the good convergence of the policy head. That: "seem to be defending its pieces" is clearly due just to the network following the policy path into the MCTS. With a minimal value head "good" convergence the network would never lose the queen in that way.

[gcp]

Yeah the second example makes it a bit obvious when white is attacking.

[kiudee]

@Zeta36 I agree that with a value weight of 0.01 it could be that our networks did not learn a good value function. It seems we have to find a good balance there.

[Zeta36]

@gcp, I don't know why you are so "aggressive" against the possibility that we could have an issue with the value head. I think the facts are there and all point out in that direction: look for example into the @Error323 MSE plots (scaling up by 4), and look how the games above seems to follow the policy head but as soon as the policy fails (after the first opening movements easy to memorize by the policy) the game turns chaotic. With a minimal value head "good" convergence the game should follow all the time a much more smooth strategy.

[gcp]

I'm objecting to the idea that if you have a large dataset with:

70% class A
20% class B
10% class C

That it's impossible to build a classifier because it will always return class A.

I don't disagree that it's hard to train the value head, but the constant implication that it is not possible (which flies in the face of established evidence, unless you believe DeepMind made things up) I just do not understand. And there is data supporting that. If it had fatally converged to 0, the network would not be returning 68% in some positions.

[kiudee]

@Error323 What is the current preferred training method after your changes?

This is what I would do now, judging from the code (before I used parse.py on the chunks directly):

1. Create chunks using lczero --supervise data.pgn
2. Convert chunks into binary format using leela_to_proto.py
3. Train using supervised_parse.py

[Zeta36]

I'm not gonna fight against you, @gcp. But I think we should study at least the possibility that the value head is not learning because of some trouble (or incapacity).

And I don't say it's impossible to train the head value but that maybe it's not possible in the way we are trying until now. And the facts are over there (MSE loss and game quality are there even you don't want to see them).

[Error323]

From @kiudee's results, the discussion here and my own experience thusfar, I think we need to train for more steps and change the learning rate decay function. Ideally we would alter the learningrate decay function such that the lr decreases when both the policy- and value head have converged as @gcp suggested in another thread. From @kiudee's graphs and mine you can clearly see that the MSE and or policy has not reached convergence during some of the lr steps and so we decrease the learning rates too soon, which in turn results in slower learning --> significant more steps.

Regarding training method, indeed steps 1, 2 and 3 are correct @kiudee. This helps speedup the trainingprocess for me by a factor of 2. The conversion of the planes to binary proto is very cpu intensive, much more so than in the Go version as we have more planes to consume per position.

[gcp]

And I don't say it's impossible to train the head value but that maybe it's not possible in the way we are trying until now. And the facts are over there (MSE loss and game quality are there even you don't want to see them).

If you read my post you'll see I don't disagree that the results aren't good, I'm objecting over the implications that it's impossible.

Another idea: train with a larger regularizer (large enough to control MSE over-fits), but drop the weighting of the policy network. See if you get better MSE performance than we have now. After this has happened, make the weights equal and go to a normal regularizer.

I will fiddle with some of these.

[Zeta36]

If you read my post you'll see I don't disagree that the results aren't good, I'm objecting over the implications that it's impossible.

I never said it's impossible, just that maybe it's impossible in the way we are currently trying. So I guess we are more or less agree.

Another idea: train with a larger regularizer (large enough to control MSE over-fits), but drop the weighting of the policy network. See if you get better MSE performance than we have now. After this has happened, make the weights equal and go to a normal regularizer.

Good idea. And there are a more lot of things we can try before going as crazy to generate more data. By the way, this is the kind of "tricks" I think DeepMind could be "hiding" in their last (and short) paper about AZ.

[gcp]

DeepMind didn't talk about doing SL in the last paper at all. We're doing it because it's interesting and useful for debugging.

[jkiliani]

Good idea. And there are a more lot of things we can try before going as crazy to generate more data. By the way, this is the kind of "tricks" I think DeepMind could be "hiding" in their last (and short) paper about AZ.

Deepmind has always been in the habit of publishing what worked for them, not what didn't work (and they're not exactly the only ones in academia doing that). So there's no alternative for our projects here to trying out multiple things and finding out what works and what doesn't, even if that results in occasional setbacks.

[Error323]

DeepMind didn't talk about doing SL in the last paper at all. We're doing it because it's interesting and useful for debugging.

Indeed, it's very likely they didn't need to because the whole purpose of that paper was to show the generic power of the algorithm. They only had to adapt the movegen and NN I/O. I think that before we start selfplay we should get a real solid understanding of the architecture through experimentation and good results in SL mode, that is, similar to the SL results of Go.

[Zeta36]

DeepMind didn't talk about doing SL in the last paper at all. We're doing it because it's interesting and useful for debugging.

Of course I know that, but a SL has to be able to train the model if a self-train pipeline is able to. So as we are not able to train the model in a correct way using SL (and huge of data) then the sel-play will not be able to do it much better.

Supposing that just more (self)data will fix this issue with the head value is very naive.

[gcp]

I'm not so sure about that. Despite that Leela Zero obviously trains well with self-play data, I didn't manage to get a good SL trained network either without messing with the MSE values. And the same was true in the DeepMind paper.

The RL data is more diverse and more specifically shows the holes in the knowledge of the current network, in a way SL data does not do.

Consider this: what do you think the draw rate is for the SL data versus the RL data?

[Zeta36]

I'm not sure about your position, @gcp. I doubt a model that fully fails in a supervised manner will do much better in a self-play data training.

Anyway if you prefer working in the brute manner playing some millions (self)games and look to see what happens instead of studying the issue theoretically. Well, it's an (expensive) option.

We'll see in some months the results ;).

[jkiliani]

@Zeta36 The whole Deepmind approach of Alphago Zero and Alpha Zero is about using self-play reinforcement learning, presumable because they got much better results from doing so than using SL only. Sure it costs resources, but it is really only self-play with MCTS and exploration from Dirichlet noise and temperature that will fix any knowledge holes in the training data set. At some point leela-chess will also have to start a self-play pipeline to figure out what really works in the chess context. Studying the issue theoretically is a good starting point but can only get you so far.

[Zeta36]

Of course, self-train is the final goal. I just comment that try self-training with a model that fully fails in a SL manner it's not a good idea.

[jkiliani]

I think "fully fails" is an unfair characterisation for the RL results by @glinscott, @Error323, @kiudee and also @Akababa and yourself in the chess-alpha-zero context. There are already multiple networks that play at a reasonable human level even if not close to the level of Stockfish. I doubt that would happen with the network size in use here anyway, but for the moment the point is to prove the setup works and will results in a network that can significantly outperform a SL net of the same dimensions. Comparing it to Stockfish or other top engines is just not realistic yet.

[Zeta36]

Fully fails in the sense it is unable to learn any chess concept or strategy and only seems to follow the well converged policy head. In this sense all the plots about the MSE showed until now have a ver strange and suspicious convergence curve (always falling quickly around ~0.6 once scale up by 4).

My personal opinion it's just we should wait for the model to be able to get a realistic head convergence in a SL manner that show us the network has learn any minimal value strategy.

But as I say it's just my opinion, if you want to start already creating millions of self-training games, it's fine. We'll see in some months if we get something or not.

[gcp]

I agree that without a value head that has at least some sensible output you can't even really be sure the thing is properly debugged.

For all you know your tree search has black and white scores reversed or something, you'd never notice.

I'm just saying that RL data has some better properties that make it easier to work with.

[Akababa]

I really don't understand all the fuss about the value head. Why can't we just remove the color plane from the input?

[kiudee]

@gcp I was processing your latest stockfish games and got an invalid game:

Processed 223910 gamesInvalid game in sftrain_clean.pgn/data

Scid also only reads 223k games.

[gcp]

They're from pgn-extract so that would be strange. I uploaded a new one this morning, this loads fine for me in Scid (vs PC).

[gcp]

I've been looking at the training, and it looks like the policy and value heads are from the AlphaGo Zero paper, not the Alpha Zero paper? That would affect things! I'll file a separate issue.

[kiudee]

Just to make sure that there were no network errors, I get the following checksum:

2386fc9cb17e88a0c0c06fce0bd50288  sftrain_clean.pgn/data

edit: Found the problem. The download using wget works and I get 352k games:

f1825d12c58d7ac04b93d80c79a62ea2  data

The problem was Firefox - it was (repeatedly) downloading an incomplete file. Any idea why?

[gcp]

Any idea why?

Initial download failed and the partially downloaded file was not deleted? (And, I think, then trying to resume after I had updated the file) If you check your downloads dir you might find .part file.

Downloading with Firefox works fine here FWIW.

[Error323]

@gcp what do you mean? I don't recall the AlphaZero paper talking about the NN architecture.

[gcp]

@Error323 They describe the policy output head. It needs to be quite different than from Go because Go moves only have to squares, which map nicely to a single board. But not so for chess. See issue #47 for more details.

It looks like here the policy output from Leela Zero was copied without making the required change.

[glinscott]

@Error323 has proven that it is indeed possible :). Congrats!