Leela-chess: Training progress gen3

[Error323]

This time I properly interleaved the chunks and we're back to our trusty sudden drop in MSE Loss. I made 4 passes through 99'080 chunks, resulting in 4'954'000 datapoints. Here are the results of two separate 50 game matches of gen3 vs gen2. And I also played it against gen1 for good measure.

Score of lc_gen3 vs lc_gen2: 48 - 1 - 1  [0.970] 50
Elo difference: 603.86 +/- nan

Score of lc_gen3 vs lc_gen2: 43 - 5 - 2  [0.880] 50
Elo difference: 346.12 +/- 172.18

Score of lc_gen3 vs lc_seed: 48 - 0 - 2  [0.980] 50
Elo difference: 676.08 +/- nan

The network is uploaded and machines are crunching towards the next generation.

[jkiliani]

@Error323 Could you please set http://162.217.248.187/networks so it allows manually downloading archived nets? I'd like to do a strength test for the new net, but the client still deletes old nets automatically so I don't have de1ddecb anymore at the moment...

[Error323]

@jkiliani let's ask @glinscott. It's not my server, I don't know what the bandwidth limitations and costs are.

[jkiliani]

Have you decided yet what you are going to use for training window size? With gen3 games, added to gen2 and seed games plus the random games you generated, we're probably going to end up above 250,000 games, which is the window Leela Zero uses. At that point, it could be useful to start windowing out the older games.

Edit: No rush about the server, you already tested it enough. But @glinscott, the .pgn viewer suddenly doesn't open for me anymore on the computer, just on Android. Was anything changed with the game viewer?

[Error323]

Have you decided yet what you are going to use for training window size? With gen3 games, added to gen2 and seed games plus the random games you generated, we're probably going to end up above 250,000 games, which is the window Leela Zero uses. At that point, it could be useful to start windowing out the older games.

Through some experimentation :) I guess a binary search through the windowsize.

[jkiliani]

I'm currently seeing a lot of black wins in the self-play on my computer... how do the initial statistics look for 9b568ab2? Did the color imbalance even out or reverse?

[Error323]

[jkiliani]

So game length continuing to shorten, draws getting even more rare, color imbalance reducing. Looks great!

[Zeta36]

Yes, this looks very promising!!

[kiudee]

I pitted this network at 100k playouts against Stockfish Level 1 on Lichess and it won for the first time:

[jkiliani]

Cool, reminds me of when Leela Zero for the first time beat IdiotBot (30k) (https://www.reddit.com/r/cbaduk/comments/7f8nsu/lz_265k_vs_idiotbot/)

Do you agree that the value head already seems pretty well differentiated now toward winning material, but the policy head still has a lot of trouble finding the right moves to capture and protect pieces?

[Error323]

Yeah a part of me thinks we should've used move encoding as described in the AlphaZero paper. Our flat representation lost spatial information from the convnets. It'll still work though and it reduces the chunk file size.

[jjoshua2]

I thought we were using the same inputs as AZ. Why aren't we? I don't know what spatial information from convents means.

[glinscott]

@Error323 Could you please set http://162.217.248.187/networks so it allows manually downloading archived nets? I'd like to do a strength test for the new net, but the client still deletes old nets automatically so I don't have de1ddecb anymore at the moment...

@jkiliani done -- you can now download the networks from the networks page.

[glinscott]

I thought we were using the same inputs as AZ. Why aren't we? I don't know what spatial information from convents means.

@jjoshua2 the inputs the AZ team used are significantly larger. They both represent the same information, the AZ paper even mentioned they used the smaller representation, it just took slightly longer to converge. I figured for us non-TPU users, it might be better to start with this one :).

[jjoshua2]

So we might run out of gpu memory with the full inputs? And were making a
compute memory tradeoff towards more compute?

On Mar 7, 2018 2:05 PM, "Gary Linscott" [e-mail] wrote:

I thought we were using the same inputs as AZ. Why aren't we? I don't know
what spatial information from convents means.

@jjoshua2 https://github.com/jjoshua2 the inputs the AZ team used are
significantly larger. They both represent the same information, the AZ
paper even mentioned they used the smaller representation, it just took
slightly longer to converge. I figured for us non-TPU users, it might be
better to start with this one :).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/glinscott/leela-chess/issues/100#issuecomment-371248172,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AO6INFc17h1_ono6jBPlSHGSv866aJq2ks5tcC9ugaJpZM4Sf9Pi
.

[glinscott]

@jjoshua2 it should be both less compute, and less memory. It's just slightly harder for the network to learn.

[jkiliani]

I'm trying to repeat @kiudee's experiment with cutechess_cli. After some experimentation, I used

./cutechess-cli -rounds 100 -tournament gauntlet -concurrency 2 -pgnout SF0.pgn \
 -engine name=lc_gen3 cmd=lczero arg="--threads=1" arg="--weights=$WDR/gen2-64x6.txt" arg="--playouts=800" arg="--noponder" arg="--noise" tc=inf \
 -engine name=sf0 cmd=stockfish_x86-64 option.Threads=1 option."Skill Level"=1 tc=40/20 \
 -each proto=uci

to set up Stockfish in a hopefully properly handicapped fashion. Does anyone here have experience in how to use a lowered skill level Stockfish with cutechess_cli? When I set tc=inf, Stockfish seemed stuck even though I expected that it would move in fixed time by setting Skill Level...

I canceled the match at 6-0 in SF's favour since I'm not sure the handicap is actually working properly, in which case the match would be pointless...

[kiudee]

Just for reference, here are the levels and time controls lichess is using:

    AI level 1: skill 3/20, depth 1, 50ms
    AI level 2: skill 6/20, depth 2, 100ms
    AI level 3: skill 9/20, depth 3, 150ms
    AI level 4: skill 11/20, depth 4, 200ms
    AI level 5: skill 14/20, depth 6, 250ms
    AI level 6: skill 17/20, depth 8, 300ms
    AI level 7: skill 20/20, depth 10, 350ms
    AI level 8: skill 20/20, depth 12, 400ms

The time control is very short and in ms per move.

[jkiliani]

I think it's probably too early for matches against other engines, at least on 800 playouts. It was likely the 100k playouts you used which gave Leela Chess enough of a boost to win against a handicapped Stockfish. I'll just try this again in a few more network generations.

[Error323]

p.s. I began training gen4 a few hours ago. using a window of 100K chunks, meaning the first rng chunks are no longer used now.

[Error323]

:scream: :flushed: I need to step up my training game, our input rate is increasing. Tomorrow night I'm gonna revamp.

[jkiliani]

The game length statistic of decisive games for gen3 is starting to look bimodal, any ideas what this could represent?

[Error323]

Could be the result of https://github.com/glinscott/leela-chess/commit/8b42445522fda80090588f91e58a579d355f7366

[killerducky]

@Error323 why does the MSE/Policy loss start over at a high value? Are you not starting from the previous best network weights?

[Error323]

Indeed, it's also part of Friday night's code night. I want to start with a high lr but with previous weights.

The goal for now is to constantly take the last 100K chunks and use the previous weights. Start at lr 0.02 and decay with 0.1 after N steps. Then upload and repeat.

[killerducky]

Ok great. I like the idea of redoing the learning rate annealing every time. I guess the initial high LR will shake the old net out of its local maximum and then as the LR lowers it can settle into a new maximum.

Edit: More wild speculation: Maybe much later it could cause problems if shaking out of old local maximums causes it to forget knowledge it learned from games that have passed out of the training window. This could be an argument for not raising the LR. As usual it's hard to know without doing a "3 day" experiment. Assuming you have 5000 TPUs. ;-) In any case I don't think it will matter at all while the network is still so weak.

[jkiliani]

I think the method used by @Error323 will do just fine even later, if the reset learning rate at the start of each training run is gradually raised when the net approaches saturation. For the early training, this idea may well be the main reason for the steep progress curve so far.

[Error323]

Ok I uploaded a new neural network, but I was impatient and so it's performance is not as great as the previous onces, but our game input rate is so high... I cracked under pressure :sweat_smile:

Score of lc_gen4 vs lc_gen3: 33 - 15 - 2  [0.680] 50
Elo difference: 130.94 +/- 104.97
Finished match

It's still a better net, but I should've trained for more steps (only 140K this time). Anyway, working on the online training version! It should bring a lot of relief :)

[jkiliani]

Not sure this is really the number of training steps... I think it rather plausible that the training window now includes such a wide range of playing strength that progress is somewhat held back by the remaining rng games.

Anyway, this net is still solid progress, thank you for your efforts! Are we choosing the always-adopt approach by the way? Sooner or later, a trained net is going to regress against the previous one, should we simply use it anyway at this point?

Edit: Results of the match I ran last night:

Score of lc_gen4 vs lc_gen3: 38 - 9 - 3  [0.790] 50
Elo difference: 230.16 +/- 122.02
Finished match

It does look somewhat better for gen4 than your match, but is easily within each other's error margins.

[amj]

@Error323 can you get your training to run fully-automatic? I.e., auto-start when there's enough new games, have the clients wait if there's too many games, etc?

(btw, first win vs level 1 -- very exciting!)

[Error323]

Anyway, this net is still solid progress, thank you for your efforts! Are we choosing the always-adopt approach by the way? Sooner or later, a trained net is going to regress against the previous one, should we simply use it anyway at this point?

@glinscott is working on the evaluation code for clients so that will be automated.

Hey @amj! it's highest on my list right now. It would be great to just be able to let it run.

[amj]

@Error323 Nice! I'm not super happy with my solution, it's kinda hacky. Probably the most important part of it is that i have it send me a SMS if the job falls over. Not sure where you're running it (locally? cloud?), but I strongly recommend some sort of monitoring of CPU usage or bandwidth or something that will trip if it dies. Especially if you're as impatient as I am!! :)

[Error323]

What makes it crash then? Are you killing/restarting the training job every time?

I'm running the training at home using a single GPU (1080ti) atm. But I have two, so I can double the speed almost. I was hoping to come up with a way that would fully automate it. No job restarting. After each training session, ingest the latest 100K chunks or so, reset the learning rates and start again from the previous network.

The problem is it requires testing and experimentation while clients are crunching. I don't want to let their cycles go-to waste :sweat_smiley: I'll start out simple and semi automatic. Improving the pipeline autonomy as we go along.

[amj]

Anything that runs for long enough will have something happen, even if they're just host maintenance events. For me, the job itself is stable but -- especially if a training run will take months -- you will eventually want to update the running code with some new thing that has bugs :)

[grolich]

@glinscott won't it actually take more compute time overall? It's less computation per eval; however, deepmind said it takes longer to converge, not that it just takes more steps.

Is there a prohibitive memory cost to the other representation?

Actually, in a different talk David Silver gave about AlphaZero, he specifically mentioned that they measured in actual time rather than just training steps. And also that the direct representation took longer to converge (well, that appeared in the paper as well).

"A little more time" to converge might be a lot more time when we aren't aided by all their TPUs.
Add to that the fact that we don't know how much "little more" time we're talking about, this may be weeks or even a lot more in our setup.

I think the idea that the current representation would be faster is based on an assumption that the process would take just a few more training steps... Which doesn't seem to be the case as far as I can see.

[jkiliani]

I think it's speculation that another representation would really be faster. Even Deepmind say:

We also tried using a flat distribution over moves for chess and shogi; the final result was almost identical although training was slightly slower.

That doesn't sound so bad, if it strongly affected learning, they would have worded this differently. In any case, if at some later point we decided to change representation, this could be done as part of a bootstrap with enforced client update. So far, what we are using seems to work very well, easily visible by the Elo progress compared to Leela Zero. And as long as it works, why change it?

[Error323]

@amj that's true :). Well I'll be hawking the system anyway and when I stop hearing its fans going I know something is wrong :smile:

And as long as it works, why change it?

We could experiment some with a lower amount of outputplanes going into each head. Currently we're at 32. This results in huge FC layers. We could try 16 or maybe even 8.

So far, what we are using seems to work very well, easily visible by the Elo progress compared to Leela Zero.

This is of course relative performance against itself. We can only know for sure if we also train the other network architecture and play them against eachother at the same level. I _think_ the other output representation converges in less trainingsteps (still requires a lot of simulations which is the bulk of the work), but we are also missing details about how many planes are going into the heads. This was not specified by DeepMind.

Back to gen4: I resumed training after uploading the current net, the rms on the value lowered and the accuracy increased some more. Then I played it (step 240K) against gen3 again, and performance was worse (losing) as if it got stuck in some local optima? Interesting...

[grolich]

@jkilliani I definitely agree that if what is currently done works we should just keep doing it.

Just wondering if a "small" difference in time for them is translated into weeks for us,
given the it only took hours for them in total, and the comparison gcp made as far as time to run on standard hardware at the start of the leela zero project (granted, there have been many optimizations since then).

Agreed though, as long as the current solution works, no reason to change it.

Just a potential issue to keep in mind for the future.

[Error323]

Generation 5:

Score of lc_gen5 vs lc_gen4: 70 - 25 - 5  [0.725] 100
Elo difference: 168.40 +/- 75.60

Trained in 30K steps using the online version, bootstrapped from gen4. Learning rate boundaries:

• lr 0.01 steps 0 ... 15K
• lr 0.002 steps 15K+1 ... 25K
• lr 0.0005 steps 25K+1 ... 30K

Training itself is just incredibly slow (this took ~16 hrs) I'm working on optimization improvements now.

[jkiliani]

Very nice progress, but I'm curious about the training speed: Did the online version reduce the speed in some way? By the way, what is the oldest data still part of the window now? Still rng, or gen1 (seed)?

[Error323]

Yes, online reduced the speed immensely as it needs to parse every chunk (unzip, decode into tensorflow format) before it can be processed by tensorflow and we're only using 1/8th of every chunk when opening. So it does this every single time. The gpu is stalling like crazy while the cpus cannot keep up. Since gen4 we're using a sliding window of 100K chunks. Which is approximately 250K games. RNG has been out of the picture for 2 gens now.

When the binary version works in online mode we'll be able to reduce the ~16 hrs to ~2.5 hrs.

Edit: the nice thing about the online version is that we'll be able to apply the symmetries and we'll make multiple passes through the chunks, utilizing our data very effectively. Which means we might be able to reduce the sliding window further and climb quicker as a result.

[jkiliani]

Since gen4 we're using a sliding window of 100K chunks. Which is approximately 250K games. RNG has been out of the picture for 2 gens now.

I can't quite follow here: We're at 210k games, out of which 180k are from the seed net or newer, and 16 hours ago it was probably around 20k less, i.e. only 160k games from trained networks. How can we use a window size of 250k games, but not have used rng games since gen4?

[Error323]

Yeah I got the 250K games from using an average of 80 plies per game. That average is incorrect :sweat_smile: sorry. I just checked, the server now has 211'268 chunks.

edit: To be clear, we still left rng 2 gens ago.

[glinscott]

Graph updated, the climb continues :). Congrats!

[jkiliani]

Statistics are still weak, but we now seem to have a slight reverse color bias: More black wins than white.

In any case, it looks like the network training itself is sufficient to rectify color imbalances all by itself, even without exploiting symmetries in training...

[jkiliani]

I made another match, gen5 against gen3:

Score of lc_gen5 vs lc_gen3: 86 - 11 - 3  [0.875] 100
Elo difference: 338.04 +/- 107.79

Consistent if slightly higher than the rating difference from @Error323's match. I also did another try to match gen5 against the weakest setting I could come up for with Stockfish:

./cutechess-cli -rounds 100 -tournament gauntlet -concurrency 2 -pgnout SF0.pgn \
 -engine name=lc_gen5 cmd=lczero arg="--threads=1" arg="--weights=$WDR/gen5-64x6.txt" arg="--playouts=800" arg="--noponder" arg="--noise" tc=inf \
 -engine name=sf_lv1 cmd=stockfish_x86-64 option.Threads=1 option."Skill Level"=1 tc=40/1 \
 -each proto=uci

No luck yet, I cancelled that match at 12 straight losses for Leela Chess. Shouldn't be much longer though until there are at least occasional draws and wins, since @kiudee demonstrated that LCZero can win against a weak Stockfish with 100k playouts.

[Error323]

@jkiliani could you try 100 games against gen4 instead of gen3? If that's consistently higher then I think my OpenCL errors are to blame. I should probably lower the concurrency at that point.

[zz4032]

There is a Level 0 in Stockfish:
option name Skill Level type spin default 20 min 0 max 20
It's about 1000 Elo based on a tournament between different level versions I did once.

[jkiliani]

@Error323 I cannot rule out the effect you describe, but at this point it seems rather unlikely to me. I think the discrepancy is more likely found in the rating of gen4 vs gen3, since I did a reference test https://github.com/glinscott/leela-chess/issues/100#issuecomment-371645511 that showed better performance for gen4, but still within the error margin of your test. Since either of our gen4 vs gen3 matches were only 50 games, the Null hypothesis seems to be the most plausible explanation here. Out of curiosity, I tried the Fisher exact test (http://www.socscistatistics.com/tests/fisher/Default2.aspx) on our test gen4 vs gen3 matches: It shows a significance level of only 0.24 that the samples are dependent, so we simply have too few games to conclude anything from this test.

There's not much point in trying to prove effects of OpenCL errors unless we're willing to invest into much better statistics than this, i.e. at least several hundred games. Personally I find it unlikely that OpenCL errors would affect strength while the nets are still so weak, but if you think it's likely, I'd recommend you compile a CPU only version of lczero, and match it against the OpenCL version using your supervised network for both sides. This should have the best chance of detecting such a bias if it's there. Comparing matches of different nets between a CPU only setup and an OpenCL setup may fail to prove the effect, since the OpenCL errors should affect the strength of both sides.

At some point we should probably use more robust statistics for the match graph, but for now I don't see the harm since every net so far we clearly stronger than the last, and we're going to end up with an inflated Elo progression in either case.

[Error323]

Ok we're experiencing some setbacks with respect to the pipeline V2 see #104. I'll train a new net in manual mode for now to make sure people stay invested.

[Error323]

Generation 6:

Score of lc_gen6 vs lc_gen5: 86 - 14 - 0  [0.860] 100
Elo difference: 315.35 +/- 105.75

Trained in 28K steps, bootstrapped from gen4 (not a typo, it was a test), using 100K chunks from gen4 and 5.

• lr 0.01 steps 0 ... 25K
• lr 0.001 steps 25K+1 ... 28K

From this I conclude that a 100K window of the previous 2 networks works very well. I guess once the online method is ready I can really experiment with the window size. This net (gen6) was trained using the offline method again, took 2hrs 8min. Online renders my system completely useless for work and takes much longer, thats why we need V2 (:

@glinscott if you would do the honors of updating the graph :) We've passed 2K elo :collision:

[kiudee]

This network is a noticeable upgrade in strength. Here is a game with 10k playouts against Lichess Stockfish Level 1:

[Zeta36]

Who is who @kiudee? is white our Leela?

[kiudee]

@Zeta36 Sorry forgot to mention. lc_gen6 is white.

Here is another game - this time _winning_ against Lichess Level 2 with 100k playouts and white:

Seems like it learned to value queens highly (look at how it is running away with her to safety rather than mate the king).

edit: Level 2 corresponds to

AI level 2: Stockfish v9 skill 6/20, depth 2, 100ms per move

[jkiliani]

I guess I have to conclude that the increased playouts make all the difference for Leela Chess. I just tried to match it against Stockfish Level 0 with 25 ms/move with its standard settings

./cutechess-cli -rounds 100 -tournament gauntlet -concurrency 2 -pgnout SF0.pgn \
 -engine name=lc_gen6 cmd=lczero arg="--threads=1" arg="--weights=$WDR/gen6-64x6-28000.txt" arg="--playouts=800" arg="--noponder" arg="--noise" tc=inf \
 -engine name=sf_lv0 cmd=stockfish_x86-64 option.Threads=1 option."Skill Level"=0 tc=40/1 \
 -each proto=uci

with the result:

Score of lc_gen6 vs sf_lv0: 0 - 16 - 1  [0.029] 17
Elo difference: -607.41 +/- nan

The one draw by the 50-move rule is more than I've ever seen so far in matches against a weakened Stockfish so I suppose that's progress already. I'll do a longer match for the next generation.

[glinscott]

@Error323 graph is updated! Congrats, that's a big jump in strength :). And only a few hours training!

[jkiliani]

After seeing gen6 draw once against Stockfish with skill level 0, I reconsidered and went through with a 100 game match to see how big the gap still is. Here it goes:

Score of lc_gen6 vs sf_lv0: 3 - 95 - 2  [0.040] 100
Elo difference: -552.08 +/- 257.07

I'll try this again with the next generation net, to see if the score against sf_lv0 narrows at a rate comparable to what the self-play Elo suggests.

Combined with the tests I posted in https://github.com/glinscott/leela-chess/issues/109#issuecomment-372701765, this suggests that gen6 is something like 1000 Elo weaker than the supervised net kbb1-64x6-796000.txt. I expect that the self-play Elo gain needed to beat it is probably more than that.

[Error323]

Score of lc_gen7 vs lc_gen6: 91 - 6 - 3 [0.925] 100
Elo difference: 436.43 +/- 143.57

%YAML 1.2
---
name: 'online-64x6'
gpu: 0

dataset:
    num_chunks: 100000
    train_ratio: 0.90
    input: '/mnt/storage/home/fhuizing/chess/distributed-dataset/run1/'

training:
    batch_size: 2048
    total_steps: 50000
    shuffle_size: 524288
    lr_values:
        - 0.01
        - 0.001
        - 0.0005
    lr_boundaries:
        - 35000
        - 45000
    policy_loss_weight: 1.0
    value_loss_weight: 1.0
    path: '/mnt/storage/home/fhuizing/chess/networks'

model:
    filters: 64
    residual_blocks: 6
...

Fully trained with the online pipeline V2! The yaml file shows the training parameters.

[jkiliani]

Wow that's great, have to do another round vs Stockfish now! And NN cache just got merged too so our game rate will go up a lot as well...

Current score:

Score of lc_gen7 vs sf_lv0: 3 - 16 - 0  [0.158] 19

The difference seems to be narrowing considerably, just not by 436 Elo. Let's see how the final result looks like...

[jkiliani]

I finished the match of lc_gen7 (800 playouts) against Stockfish Level 0, at 1 sec/40 moves time control. The new network cut Stockfish's margin by a lot, more than my last post suggested:

Score of lc_gen7 vs sf_lv0: 23 - 75 - 2  [0.240] 100
Elo difference: -200.24 +/- 81.11

The Elo difference decrease to https://github.com/glinscott/leela-chess/issues/100#issuecomment-372798753 does not quite match @Error323's measured 436 Elo gain, but it's not that far off. I think the next network against sf_lv0 will be really interesting. After this it may already be time to increase Stockfish's skill level...

[jjoshua2]

@Error323 does this mean we are close to being automated? Since each training has such a big gain it means it would be good to do it even every 10k games it seems.

[glinscott]

@Error323 awesome work! Graph updated on main page. I really need to finish the match support at this point. It's getting closer, but a fair bit of code, as can be seen here: https://github.com/glinscott/leela-chess/compare/match_support.

[Error323]

@jjoshua2 yes, but not quite there yet. As @glinscott said, the match support code still requires a bit of work. And I also want to build up some more confidence with the online training before letting it run without looking :see_no_evil:

Regarding the new gen, I made a mistake :sweat_smile: had to redo the training so it's gonna take a bit longer.

[Error323]

Generation 8:

Score of lc_gen8 vs lc_gen7: 82 - 14 - 4  [0.840] 100
Elo difference: 288.06 +/- 94.51
Finished match

[Zeta36]

Again a big step forward. Congrats!

[jkiliani]

Congrats on the training, but I'm afraid I have some (still tentative) bad news: I ran the new net against Stockfish Level 0, expecting it to win narrowly. To my surprise, so far:

Score of lc_gen8 vs sf_lv0: 9 - 24 - 0  [0.273] 33

Obviously the statistics here are still very weak, but it doesn't look like the self-play rating transfers to an external measurement this time. I'll run the whole 100 matches and report back then.

If the current trend is confirmed, we may be experiencing the self-play rating inflation, a phenomenon that Leela Zero experienced when validating its progress curve on http://www.yss-aya.com/cgos/19x19/standings.html. An even more extreme example was the 9x9 Go experiment by @Alderi-Tokori, who trained a long sequence of nets, each of which beat their predecessors easily, while matches against external bots confirmed that progress had actually stalled (https://github.com/gcp/leela-zero/issues/863#issuecomment-372243689).

If I can confirm these results, one thing we could try is training on a larger window. While training on a small window tends to find nets that beat the previous one more easily, it seems to also exacerbate the self-play rating inflation.

[killerducky]

LZGo has a ratio of about 3:1 self play Elo to CGOS Elo. Other Go programmers reported similar trends.

[Error323]

Currently playing against gen6 and 5 after your post and it's quite convincing so far. I've seen games where it got ahead with 13-0 and still came out equal strength, I think waiting is important :)

Score of lc_gen8 vs lc_gen6: 30 - 0 - 0  [1.000] 30
Score of lc_gen8 vs lc_gen5: 18 - 0 - 0 [1.000] 18

edit: added gen5. @jkiliani it could just have learned something smart given it's previous self which is trivial for stockfish l0. E.g. "rook is important".

[jjoshua2]

Looking at the play (too small sample size I know), I felt it was a little stronger with a lot more pawn captures, and not missing mate in 1s, and a lot of times when a piece was left hanging it was immediately captured, but still very bad since a lot free pieces were not taken. I was excited when it did a fork, but then didn't capture.

[jkiliani]

@Error323 Sounds plausible... Likely, it's a good idea to promote even if a match fails for that reason, since Deepmind adopted that procedure for Alpha zero.

[glinscott]

@Error323 great stuff! Really exciting to see the progress march forward. Now, to get the match support in... It's getting closer!

[jhellis3]

The Elo inflation is not so much a self-play issue as it is a single-opponent issue. The magnitude of the inflation will also decrease significantly over time as overall strength improves.

I am not sure promoting regardless of results is the best way to go. Another option would be to play a gauntlet vs self (current best) and the two previous networks (regardless of if they passed). This would limit the self-play games to 1/3rd, which should reduce the over-fitting/inflation issue.

[jkiliani]

I now have the result of the match against Stockfish Level 0:

Score of lc_gen8 vs sf_lv0: 34 - 65 - 1  [0.345] 100
Elo difference: -111.37 +/- 72.46

There was after all a modest gain compared to the last net, but we probably just have to expect self-play Elo to correspond to roughly 1/3 Bayes Elo with multiple, varied opponents. The type of gauntlet run by @CMCanavessi seems to be the best way to do that, I just don't have the resources for something on this scale. I'll continue the matches with handicapped Stockfish to see how the progression there corresponds to self-play Elo and to @CMCanavessi's tournament.

[jjoshua2]

[White "lczero gen8 100k"]
[Black "stockfish_9 lvl 2"]
100k playouts vs skill level 2. TC set to 1 ply but it only seemed to limit SF, which was fine since it was still a close game. Tried to get chesstempo but couldn't figure out how to link it.

```

1. Nc3 {+0.12/24 52s} d5 {+0.72/1 0.001s} 2. a4 {+0.18/25 54s} e5 {+1.86/1 0s}
2. Rb1 {+0.17/25 60s} Nf6 {+2.64/1 0s} 4. e3 {+0.09/25 58s} c6 {+2.03/1 0.001s}
3. d3 {+0.07/25 64s} Bg4 {+2.55/1 0s} 6. f3 {+0.50/25 65s} Bc8 {+2.05/1 0s}
4. Nge2 {+0.50/25 65s} Nbd7 {+2.08/1 0s} 8. Ng3 {+0.64/25 62s} Bb4 {+2.38/1 0s}
5. Bd2 {+0.48/25 57s} h5 {+2.40/1 0s} 10. h4 {+0.58/25 67s} d4 {+2.13/1 0s}
6. exd4 {+1.45/25 67s} exd4 {+1.21/1 0.001s} 12. Nce2 {+1.46/25 65s}
   Bd6 {+0.87/1 0.001s} 13. Nf5 {+1.44/25 59s} Ne5 {-1.52/1 0.001s}
7. Nxd6+ {+1.42/25 63s} Qxd6 {+0.49/1 0s} 15. b4 {+1.42/25 62s}
   Bd7 {+0.85/1 0s} 16. Nxd4 {+1.63/25 64s} Nxd3+ {+3.20/1 0s}
8. Bxd3 {+2.32/25 68s} Qxd4 {-0.35/1 0.001s} 18. Qe2+ {+2.26/25 65s}
   Kf8 {-0.35/1 0s} 19. c3 {+2.35/25 64s} Qd6 {+1.88/1 0s} 20. Rd1 {+2.53/25 67s}
   Re8 {+4.56/1 0s} 21. Be3 {+1.38/25 68s} Qb8 {+3.81/1 0s} 22. b5 {+2.05/25 69s}
   Nd5 {+3.88/1 0.001s} 23. Bc5+ {-1.32/25 66s} Kg8 {+4.07/1 0s}
9. Be4 {-1.14/25 67s} b6 {+2.30/1 0.001s} 25. Bd4 {+1.08/25 65s}
   Rh6 {+0.52/1 0.001s} 26. Kf1 {+1.89/25 69s} c5 {+0.71/1 0s}
10. Bxg7 {+2.10/25 67s} Nf4 {-1.68/1 0.001s} 28. Qd2 {+1.95/25 65s}
    Bxb5+ {-0.11/1 0.001s} 29. axb5 {+2.25/25 52s} Kxg7 {-0.11/1 0.001s}
11. g3 {+2.25/25 66s} Rxe4 {-0.64/1 0.001s} 31. fxe4 {+2.86/25 63s}
    Ne6 {-0.86/1 0.001s} 32. Qd7 {+2.94/25 62s} Qxg3 {+2.87/1 0.001s}
12. Ke2 {+2.66/25 63s} Rg6 {+6.30/1 0.001s} 34. Rdg1 {+2.97/24 54s}
    Qf4 {+2.35/1 0.001s} 35. Rxg6+ {+3.23/25 56s} Kxg6 {+0.15/1 0s}
13. Qc6 {+3.26/25 57s} Qg3 {+0.90/1 0.001s} 37. Ra1 {+5.14/24 43s}
    Kg7 {+1.47/1 0.001s} 38. Rxa7 {+5.33/24 57s} Qh3 {+1.22/1 0.001s}
14. e5 {+6.57/24 58s} Qf5 {-0.38/1 0.001s} 40. Qd6 {+6.47/24 46s}
    Qc2+ {-0.26/1 0.001s} 41. Kf1 {+6.43/25 50s} Qb1+ {-0.30/1 0.001s}
15. Kf2 {+7.00/24 32s} Qf5+ {-0.67/1 0.001s} 43. Ke3 {+6.70/24 49s}
    Nf4 {+0.01/1 0.001s} 44. Qf6+ {+14.53/23 30s} Qxf6 {-5.50/1 0.001s}
16. exf6+ {+14.60/23 2.6s} Kh6 {-5.50/1 0s} 46. Kxf4 {+16.91/23 25s}
    c4 {-6.30/1 0s} 47. Kf5 {+26.88/20 2.5s} Kh7 {-7.43/1 0s}
17. Ra8 {+34.46/18 0.52s} Kh6 {-M2/1 0s} 49. Rh8# {+39.10/16 0.29s, White mates}
    1-0

[White "lczero gen8 100k"]
[Black "Ruffian_105"]
[TimeControl "1 ply/move"]
[Result "1-0"]

1.Nc3 e5 2.Nf3 Nc6 { A00: Van Geet Opening: Reversed Nimzowitsch Variation, 2.Nf3 Nc6 } 3.Rb1 { (+0.23 → -0.84) Inaccuracy. A better move was 3. e4. } ( 3.e4 3...Nf6 4.Bc4 4...Bc5 5.d3 5...O-O 6.O-O 6...h6 7.Nd5 7...a6 ) 3...d5 4.e4 Nf6 { (-1.21 → +0.25) Mistake. The best move was 4... dxe4. } ( 4...dxe4 5.Nxe4 5...f5 6.Nc3 6...e4 7.Ng1 7...Nf6 8.Bb5 8...Bc5 9.d3 ) 5.exd5 Nxd5 6.d3 { (+0.34 → -0.35) Inaccuracy. A better move was 6. Bb5. } ( 6.Bb5 6...Nxc3 7.bxc3 7...Bd6 8.O-O 8...O-O 9.Re1 9...Be6 10.d4 10...exd4 ) 6...Bc5 7.a4 O-O 8.Ne4 Bb4+ 9.c3 Ba5 10.b4 Bb6 { (-0.41 → +1.09) Mistake. The best move was 10... Nxc3. } ( 10...Nxc3 11.Nxc3 11...Nxb4 12.Bd2 12...Bf5 13.Qb3 13...Nxd3+ 14.Bxd3 14...Qxd3 15.Rb2 ) 11.b5 { (+1.12 → -0.45) Mistake. The best move was 11. a5. } ( 11.a5 11...Bxf2+ 12.Kxf2 12...f5 13.Bg5 13...Qe8 14.Qb3 14...Qf7 15.Nc5 15...b6 ) 11...Na5 12.Qc2 Re8 { (-0.49 → +0.66) Inaccuracy. A better move was 12... c5. } ( 12...c5 13.Be2 13...f5 14.Ng3 14...Qd6 15.O-O 15...Nf4 16.Re1 16...c4 17.dxc4 ) 13.Ke2 { (+0.58 → -3.68) Blunder. The best move was 13. c4. } ( 13.c4 13...c6 14.cxd5 14...cxd5 15.Ng3 15...f5 16.Be3 16...Bxe3 17.fxe3 17...f4 ) 13...Bg4 { (-3.74 → -2.62) Inaccuracy. A better move was 13... f5. } ( 13...f5 14.Ke1 14...fxe4 15.dxe4 15...Nf4 16.h4 16...Bg4 17.Ng5 17...Qf6 18.f3 ) 14.Bd2 { (-2.37 → -3.72) Inaccuracy. A better move was 14. h3. } ( 14.h3 14...Bxf3+ 15.gxf3 15...f5 16.Bg5 16...Nf4+ 17.Kd1 17...Qd5 18.c4 18...Qd4 ) 14...Qh4 { (-4.22 → -1.69) Blunder. The best move was 14... f5. } ( 14...f5 15.h3 15...Bxf3+ 16.gxf3 16...fxe4 17.dxe4 17...Qh4 18.Rh2 18...Nf4+ 19.Bxf4 ) 15.h3 Rad8 { (-2.12 → -0.06) Blunder. The best move was 15... Nf4+. } ( 15...Nf4+ 16.Ke1 16...Bxf3 17.gxf3 17...f5 18.Ng3 18...Rad8 19.Qa2+ 19...Kh8 20.Qc2 ) 16.hxg4 Qxh1 17.Ng3 { (0.00 → -2.20) Mistake. The best move was 17. c4. } ( 17.c4 17...Nf4+ 18.Bxf4 18...exf4 19.c5 19...Qh6 20.cxb6 20...cxb6 21.Ke1 21...Qg6 ) 17...Nf4+ 18.Kd1 Qh6 19.g5 Qe6 20.d4 { (-1.98 → -3.06) Inaccuracy. A better move was 20. Ne4. } ( 20.Ne4 20...c5 21.c4 21...Qd7 22.Bxf4 22...exf4 23.Kc1 23...Bc7 24.Kb2 24...b6 ) 20...Qg4 { (-3.30 → -1.56) Mistake. The best move was 20... c5. } ( 20...c5 21.Nxe5 21...cxd4 22.Bxf4 22...dxc3+ 23.Ke1 23...Bd4 24.Bd3 24...Bxe5 25.Bxe5 ) 21.Kc1 exd4 22.cxd4 Re6 23.Kb2 Red6 { (-1.93 → +0.10) Blunder. The best move was 23... g6. } ( 23...g6 24.Rc1 24...c6 25.Kb1 25...Nd5 26.Bd3 26...Re7 27.Qa2 27...Rc7 28.Be4 ) 24.Ne4 { (+0.23 → -2.16) Mistake. The best move was 24. Qf5. } ( 24.Qf5 24...Qxf5 25.Nxf5 25...Ne6 26.Nxd6 26...Rxd6 27.Re1 27...Rd8 28.Bc3 28...c5 ) 24...Rd5 25.Bc3 Nxg2 26.Ne5 Qh3 { (-1.75 → -1.04) Inaccuracy. A better move was 26... Rxe5. } ( 26...Rxe5 27.dxe5 27...Nf4 28.Re1 28...Qf3 29.Be2 29...Nxe2 30.Qxe2 30...Qf5 31.Rd1 ) 27.Ng3 { (-1.39 → -4.91) Mistake. The best move was 27. g6. } ( 27.g6 27...hxg6 28.Ng5 28...Qh4 29.Nexf7 29...Rf5 30.Nxd8 30...Rxf2 31.Be2 31...Nf4 ) 27...Bxd4 28.Bxd4 Rxd4 29.Qxc7 Rb4+ { (-5.15 → -0.24) Blunder. The best move was 29... Qe6. } ( 29...Qe6 30.Qxf7+ 30...Qxf7 31.Nxf7 31...Kxf7 32.Bxg2 32...Rb4+ 33.Ka1 33...Rxa4+ 34.Kb2 ) 30.Ka3 Rxb1 { (+0.77 → ♔ Mate in 1) Blunder. The best move was 30... Rbd4. } ( 30...Rbd4 31.Qxf7+ 31...Kh8 32.Bxg2 32...Qh4 33.Be4 33...Rxe4 34.Nxe4 34...Qxe4 35.Qc7 ) 31.Qxd8# 1-0```

[Uriopass]

Here is the gif of the game for thoses who don't have the patience.

[Zeta36]

Wonderful game of Leela. There is no doubt the model is learning chess strategies.

[jjoshua2]

Here is 100k playouts against full sf9, but still limited to 1 ply. @Uriopass Note there was a game against Ruffian in the last post as well. SF won all 3 in a row no cherry picking data. I guess it's time to try ply=2
[White "lczero gen8 100k"]
[Black "stockfish_9 ply 1"]
[Result "1-0"]

1. Nc3 {+0.12/24 50s} d5 {+0.72/1 0.001s} 2. a4 {+0.18/25 55s} e5 {+1.86/1 0s}
3. Rb1 {+0.17/25 58s} d4 {+2.64/1 0s} 4. Nf3 {-0.05/25 61s} dxc3 {+2.89/1 0s}
5. Nxe5 {-0.68/25 64s} Qd4 {+4.16/1 0s} 6. Nf3 {-0.36/25 67s}
cxd2+ {+2.84/1 0.001s} 7. Bxd2 {-0.10/25 62s} Qxa4 {+3.23/1 0s}
8. b3 {-0.12/25 62s} Qe4 {+3.61/1 0s} 9. e3 {-0.06/25 64s} Nf6 {+3.44/1 0s}
10. Ng5 {-0.42/25 55s} Qe5 {+3.72/1 0s} 11. h4 {-0.57/25 67s} h6 {+4.28/1 0s}
12. f4 {-1.05/25 64s} Qc5 {+4.77/1 0s} 13. b4 {-0.88/25 66s} Qd5 {+4.75/1 0s}
14. Nf3 {-0.50/25 66s} Ne4 {+4.75/1 0s} 15. c3 {-0.69/25 62s} Bg4 {+5.85/1 0s}
16. Qc2 {-1.41/25 64s} Be7 {+6.00/1 0.001s} 17. c4 {+0.10/25 65s}
Qe6 {+5.24/1 0.001s} 18. Nd4 {+0.19/25 56s} Qg6 {+4.55/1 0s}
19. Bd3 {-0.27/25 65s} f5 {+3.14/1 0.001s} 20. b5 {-0.15/25 63s}
Bh5 {+5.91/1 0s} 21. Nxf5 {+0.45/25 65s} Nc5 {+2.50/1 0.001s}
22. Be4 {+0.48/25 64s} Qg4 {+4.62/1 0s} 23. Bf3 {+3.49/25 63s}
Bxh4+ {-3.86/1 0s} 24. Rxh4 {+6.38/25 52s} Qxh4+ {-6.41/1 0.001s}
25. Nxh4 {+6.49/25 57s} Bf7 {-6.50/1 0s} 26. f5 {+6.73/25 56s} Nbd7 {-6.07/1 0s}
27. Ng6 {+6.62/25 65s} Bxg6 {-6.12/1 0s} 28. fxg6 {+8.31/25 60s}
Ne5 {-5.55/1 0s} 29. Bxb7 {+8.62/25 61s} Nxb7 {-3.42/1 0s}
30. Qe4 {+9.24/25 51s} O-O-O {-6.78/1 0s} 31. Qxe5 {+9.39/25 156s}
Nd6 {-6.67/1 0s} 32. Ba5 {+9.30/25 173s} Rd7 {-6.56/1 0s}
33. c5 {+10.15/25 189s} Re8 {-6.65/1 0s} 34. Qh5 {+10.21/25 177s}
Ne4 {-6.41/1 0.001s} 35. Bb4 {+10.18/25 171s} Kb8 {-6.50/1 0s}
36. Qf5 {+10.01/25 166s} Ree7 {-6.57/1 0s} 37. c6 {+15.50/24 160s}
Nd6 {-9.90/1 0.001s} 38. Qxd7 {+16.31/24 117s} Rxd7 {-9.52/1 0s}
39. cxd7 {+16.47/24 150s} Nb7 {-6.72/1 0s} 40. Be7 {+16.54/24 158s}
c6 {-8.68/1 0s} 41. bxc6 {+21.40/24 117s} Kc7 {-14.19/1 0s}
42. cxb7 {+21.52/24 123s} Kxd7 {-16.69/1 0.001s} 43. b8=Q {+22.07/24 47s}
Kxe7 {-16.69/1 0.001s} 44. Qe5+ {+23.57/24 76s} Kd7 {-17.42/1 0.001s}
45. Qxg7+ {+24.40/24 43s} Kc8 {-17.56/1 0s} 46. Qxa7 {+33.28/22 29s}
h5 {-M2/1 0s} 47. Rb8# {+38.97/20 6.6s, White mates} 1-0

[Uriopass]

http://www.apronus.com/chess/puzzle/editor.php

[Error323]

Sorry people, tomorrow there will be a new version. I'm tracking a bug.

Which I introduced myself :cry:

[CMCanavessi]

No hurry, better slow but solid.

[jjoshua2]

gen8-100k v sf9-2ply. SF won with kind of a long queen and rook checkmate in the corner.

[Error323]

Score of lc_gen9 vs lc_gen8: 61 - 34 - 5  [0.635] 100
Elo difference: 96.19 +/- 69.82

Still an improvement, but not as big as I had hoped. Slowdown of training caused by #119. I uploaded the net from step 48'000.

[jjoshua2]

@Error323 Are you saying that more training would likely have made a stronger net? Or just there was a long time between gen 8 and gen 9. Hopefully gen 10 will be less of a problem and also stronger!

Played gen 9 100k against Ruffian ply 2 and it lost fast 0-1. I'm wondering if Na3 is common because bishop takes and pawn retakes, like in this game, and bishops are worth a little more?

1. Na3 {0.00/25 163s} e5 {-0.27/2 0.001s} 2. h4 {+0.02/25 181s}
Bxa3 {+0.03/2 0.001s} 3. bxa3 {+0.25/25 170s} d5 {-0.17/2 0s}
4. Rb1 {+0.17/25 168s} Nf6 {-0.08/2 0.001s} 5. d3 {+0.17/25 53s}
Nc6 {-0.01/2 0.001s} 6. e3 {+0.35/25 61s} O-O {+0.16/2 0.010s}
7. Ne2 {+0.61/25 59s} Qd6 {+0.21/2 0s} 8. Rg1 {+1.03/25 59s}
Ng4 {+0.31/2 0.001s} 9. d4 {+1.29/25 65s} Qf6 {+2.01/2 0.002s}
10. f3 {+1.20/25 65s} Nh2 {+0.56/2 0.002s} 11. c4 {+1.07/25 63s}
Qxh4+ {+3.05/2 0.010s} 12. Kd2 {+0.65/25 69s} exd4 {+3.04/2 0.006s}
13. cxd5 {+1.04/25 66s} Nxf1+ {+3.23/2 0.004s} 14. Rxf1 {+0.75/25 68s}
dxe3+ {+3.28/2 0.001s} 15. Kxe3 {+0.36/25 68s} Re8+ {+4.04/2 0.001s}
16. Kd2 {0.00/25 67s} Nd4 {+4.16/2 0.002s} 17. Ng1 {+0.08/25 66s}
Bf5 {+3.47/2 0.002s} 18. Rb2 {-1.57/25 66s} Rad8 {+3.96/2 0.002s}
19. Kc3 {-2.11/25 59s} Rxd5 {+3.78/2 0.002s} 20. Rxb7 {-2.49/25 60s}
Nb5+ {+5.49/2 0.003s} 21. Kb3 {-6.00/25 58s} Rd3+ {+6.76/2 0.003s}
22. Qxd3 {-7.69/25 60s} Bxd3 {+6.54/2 0.002s} 23. Rd1 {-7.64/25 58s}
Qc4+ {+M3/1 0s} 24. Kb2 {-38.67/17 1.1s} Qc3# {+M1/1 0s, Black mates} 0-1

[Error323]

@Error323 Are you saying that more training would likely have made a stronger net? Or just there was a long time between gen 8 and gen 9. Hopefully gen 10 will be less of a problem and also stronger!

Just a long time between. I'm also running experiments to find optimizations, I may have some interesting results. But the downside is it'll take longer for new nets to arrive. Once #119 is merged, I can do both at the same time again. And once match evaluation is done by @glinscott we can iterate faster.

[Error323]

That said, is there anyone around here that can help me on the Windows side with #119? Could one of you please get the AppVeyor build https://ci.appveyor.com/project/glinscott/leela-chess/build/90. And run the client with the gen8 network (1aa02cb1).

./lczero --weights=1aa02cb1 --seed=1 --start='train 1 1' -t1 -q

And once done, upload the training.0.gz file in the data-1 directory here.

[CMCanavessi]

I can do that in 20 minutes when a 40/40 game is finished.

[jjoshua2]

I'll try. DL the file https://ci.appveyor.com/api/buildjobs/nxu1vkp3gyhwxol9/artifacts/build%2FRelease%2Flczero.exe

[Error323]

It's important to get the exe from that specific branch #119 and run it with the exact arguments shown:

./lczero --weights=1aa02cb1 --seed=1 --start='train 1 1' -t1 -q

Also make sure that before you start the data-1 directory is empty or non existent.

[jjoshua2]

I got that link from your link and clicking the top one that's not CPU only. It's ran. training.0.gz

[CMCanavessi]

I get the following:

F:\Chess\LCZero\test>lczero.exe --weights=1aa02cb1 --seed=1 --start='train 1 1' -t1 -q
Unrecognized argument: 1
Unrecognized argument: 1'
LCZero Copyright (C) 2017  Gary Linscott
Based on:Leela Chess Copyright (C) 2017 benediamond
Leela Zero Copyright (C) 2017  Gian-Carlo Pascutto
Stockfish Copyright (C) 2017  Tord Romstad, Marco Costalba, Joona Kiiski, Gary Linscott
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see the COPYING file for details.

Allowed options:
  -h [ --help ]                Show commandline options.
  -t [ --threads ] arg (=2)    Number of threads to use.
  -p [ --playouts ] arg        Weaken engine by limiting the number of
                               playouts. Requires --noponder.
  -r [ --resignpct ] arg (=10) Resign when winrate is less than x%.
  -n [ --noise ]               Apply dirichlet noise to root.
  --randomize                  Randomize move selection at root (only useful
                               for training).
  -s [ --seed ] arg            Random number generation seed.
  -w [ --weights ] arg         File with network weights.
  -l [ --logfile ] arg         File to log input/output to.
  -q [ --quiet ]               Disable all diagnostic output.
  --noponder                   Disable thinking on opponent's time.
  --start arg                  Start command {train, bench}.
  --supervise arg              Dump supervised learning data from the pgn.
  --gpu arg                    ID of the OpenCL device(s) to use (disables
                               autodetection).
  --full-tuner                 Try harder to find an optimal OpenCL tuning.
  --tune-only                  Tune OpenCL only and then exit.


F:\Chess\LCZero\test>

[CMCanavessi]

Changed the single quotes to double quotes and looks to be working

[Error323]

Ah excellent :)

[jjoshua2]

I used powershell instead of command prompt and didn't have issues.

[Error323]

@jjoshua2 could you post the PGN?

[CMCanavessi]

training.0.gz

Here's the generated file

[Error323]

Could you also post the PGN? :)

[jjoshua2]

@Error323 I uploaded the gz two posts ago. I don't see a pgn, just a training.0 instead of a gz.
Edit I guess its in my window still...

[CMCanavessi]

Full output, with PGN at the bottom:

F:\Chess\LCZero\test>lczero.exe --weights=1aa02cb1 --seed=1 --start="train 1 1" -t1 -q
Created dirs data-1
info depth 16 nodes 800 nps 677 score cp 15 winrate 50.61% time 1170 pv d2d3 g8f6 g1f3 d7d6 e2e4
info depth 16 nodes 800 nps 800 score cp -3 winrate 49.86% time 990 pv a7a5 g1f3 g8f6 b1c3 f6g4
info depth 16 nodes 800 nps 769 score cp -6 winrate 49.76% time 1030 pv g1f3 g8f6 b1a3 f6g4
info depth 16 nodes 800 nps 769 score cp -11 winrate 49.56% time 1030 pv g8f6 a2a4 d7d6 b1c3
info depth 17 nodes 800 nps 784 score cp -13 winrate 49.47% time 1010 pv b1c3 d7d6 a2a4 f6g4
info depth 17 nodes 800 nps 769 score cp 4 winrate 50.18% time 1030 pv f6g4 e2e4 g4h2 f3h2 d7d6
info depth 17 nodes 800 nps 769 score cp 22 winrate 50.88% time 1030 pv h2h3 d7d6 h3g4 c8g4 h1g1 g4f3 e2f3
info depth 17 nodes 800 nps 800 score cp -28 winrate 48.91% time 990 pv g4f6 a2a4 b7b6 f3g5
info depth 17 nodes 800 nps 761 score cp 6 winrate 50.25% time 1040 pv a2a4 b8c6 c3b5 c6b4
info depth 17 nodes 800 nps 800 score cp -29 winrate 48.87% time 990 pv b8c6 e2e4 c6b4 e4e5
info depth 17 nodes 800 nps 800 score cp 10 winrate 50.41% time 990 pv a1b1 c6b4 e2e4 f6e4 d3e4
info depth 16 nodes 800 nps 754 score cp -55 winrate 47.86% time 1050 pv d7d5 c1d2 d5d4 c3b5
info depth 17 nodes 800 nps 754 score cp 62 winrate 52.39% time 1050 pv c1g5 h7h6 g5f6 g7f6 e2e4
info depth 17 nodes 800 nps 769 score cp -39 winrate 48.48% time 1030 pv h7h6 g5f4 d5d4 c3b5 c6b4
info depth 17 nodes 800 nps 800 score cp -1 winrate 49.93% time 990 pv g5h4 c8d7 e2e3 g7g5 h4g5
info depth 17 nodes 800 nps 909 score cp -23 winrate 49.10% time 870 pv d5d4 e2e4 d4c3 b2c3 b7b6 e4e5
info depth 17 nodes 800 nps 824 score cp 0 winrate 50.04% time 960 pv e2e4 d4c3 b2c3 b7b6 e4e5 c8b7
info depth 17 nodes 800 nps 824 score cp -7 winrate 49.72% time 960 pv d4c3 b2c3 c8h3 g2h3 f6e4 d3e4 e7e6
info depth 17 nodes 800 nps 952 score cp -23 winrate 49.09% time 830 pv b2c3 g7g5 f3g5 h6g5 h4g5 h8g8 g5f6 e7f6
info depth 17 nodes 800 nps 4210 score cp -2 winrate 49.89% time 180 pv e7e5 h4g3 h6h5 f3e5
info depth 17 nodes 800 nps 776 score cp -22 winrate 49.15% time 1020 pv g2g4 g7g5 h4g5 h6g5 f3g5 c8g4 h3g4
info depth 17 nodes 800 nps 792 score cp 14 winrate 50.56% time 1000 pv g7g5 f3g5 h6g5 h4g5 c8g4 g5f6 g4d1 f6d8
info depth 17 nodes 800 nps 833 score cp -125 winrate 45.20% time 950 pv h4g3 c8d7 f3e5 c6e5 g3e5 d7a4 e5f6 d8f6
info depth 17 nodes 800 nps 1025 score cp 79 winrate 53.05% time 770 pv f6g4 h3g4 c8g4 g3e5 c6e5 f3e5
info depth 17 nodes 800 nps 833 score cp -78 winrate 46.97% time 950 pv h3g4 c8g4 h1g1 g4f3 d1f3 g5g4 f3g4 f8e7
info depth 17 nodes 800 nps 987 score cp 107 winrate 54.13% time 800 pv c8g4 b1b7 g4f3 d1f3 f8e7 f3d1 g5g4 g3e5
info depth 17 nodes 800 nps 2758 score cp -155 winrate 44.06% time 280 pv f1e2 h8g8 f3e5 c6e5 g3e5 g4e2
info depth 17 nodes 800 nps 851 score cp 98 winrate 53.78% time 930 pv f7f6 f3e5 c6e5 e2g4 e5g4 d1g4 c7c6
info depth 17 nodes 800 nps 869 score cp -44 winrate 48.29% time 910 pv f3g5 g4e2 d1e2 f6g5 e2h5 e8e7 g3e5 c6e5
info depth 17 nodes 800 nps 1379 score cp 49 winrate 51.89% time 570 pv g4e2 d1e2 f6g5 f2f3 h8g8 g3e5 c6e5 b1d1
info depth 17 nodes 800 nps 2285 score cp -130 winrate 45.01% time 340 pv d1e2 f6g5 h1g1 h8g8 g3e5 c6e5 b1d1 f8c5
info depth 17 nodes 800 nps 2162 score cp 89 winrate 53.44% time 360 pv f6g5 f2f3 h8g8 b1b7 f8e7 g3e5
info depth 17 nodes 800 nps 2285 score cp -154 winrate 44.08% time 340 pv f2f3 h8g8 b1b7 f8e7 g3e5 c6e5
info depth 17 nodes 800 nps 792 score cp 155 winrate 55.94% time 1000 pv h6h5 b1b2 h8g8 g3e5 c6e5
info depth 17 nodes 800 nps 808 score cp -148 winrate 44.32% time 980 pv e2g2 h5h4 g3h4 g5h4 b1d1 h4h3 g2h3
info depth 17 nodes 800 nps 833 score cp 120 winrate 54.60% time 950 pv d8b8 h1g1 g5g4 f3g4
info depth 17 nodes 800 nps 754 score cp -103 winrate 46.05% time 1050 pv e1e2 f8e7 g3e5 c6e5 d3d4
info depth 17 nodes 800 nps 761 score cp 184 winrate 57.03% time 1040 pv h5h4 e2e3 h4h3 g2h3 h8h3 h1h3
info depth 17 nodes 800 nps 879 score cp -155 winrate 44.06% time 900 pv g3e1 f8h6 e1h4 g5h4 h1h4 h8g8 h4h6
info depth 17 nodes 800 nps 963 score cp 117 winrate 54.52% time 820 pv f8h6 e1h4 g5h4 h1h4 h6g7 h4h1 h8h1
info depth 17 nodes 800 nps 941 score cp -134 winrate 44.83% time 840 pv e1h4 g5h4 h1h4 h8g8 h4h6 g8g2 e2e3
info depth 17 nodes 800 nps 800 score cp 169 winrate 56.50% time 990 pv g5h4 h1h4 b8a7 h4h6 h8h6 d3d4 e5d4 c3d4
info depth 17 nodes 800 nps 4000 score cp -208 winrate 42.07% time 190 pv h1h4 b8a7 h4h6 h8h6 d3d4 e5d4 c3d4
info depth 17 nodes 800 nps 1081 score cp 189 winrate 57.24% time 730 pv h6g7 h4h8 g7h8 g2g6 e8e7 b1b2 c6d8
info depth 17 nodes 800 nps 1095 score cp -192 winrate 42.65% time 720 pv h4g4 g7f6 g2g3 h8g8 g4g8 e8d7
info depth 17 nodes 800 nps 1066 score cp 122 winrate 54.69% time 740 pv g7f6 g2g1 b8d8 g1g3 h8f8 g3h4
info depth 17 nodes 800 nps 1481 score cp -188 winrate 42.79% time 530 pv g2g1 b8d8 g1g3 f6e7 g4g7
info depth 17 nodes 800 nps 824 score cp 230 winrate 58.78% time 960 pv c6e7 g4g2 c7c6 b1b2 h8f8 b2b1
info depth 17 nodes 800 nps 1269 score cp -293 winrate 38.91% time 620 pv g4g2 c7c6 b1b2 h8f8 b2b1
info depth 17 nodes 800 nps 1142 score cp 264 winrate 60.04% time 690 pv h8h3 b1b2 c7c6 g1f2 b8c7 f2g1
info depth 17 nodes 800 nps 1012 score cp -301 winrate 38.59% time 780 pv b1f1 c7c6 g1f2 b8c7 f2g1 h3h4
info depth 17 nodes 800 nps 824 score cp 239 winrate 59.11% time 960 pv c7c6 g1f2 b8c7 f2g1 h3h4 g2g3
info depth 17 nodes 800 nps 879 score cp -308 winrate 38.37% time 900 pv e2e1 h3h4 e1f2 h4h2 g1h2
info depth 17 nodes 800 nps 919 score cp 326 winrate 62.29% time 860 pv b8a7 e1e2 a7g1 f1g1 h3h4 g2g3 h4h2 e2d1 f6h4
info depth 16 nodes 800 nps 1600 score cp -388 winrate 35.47% time 490 pv e1d1 a7g1 g2g1 h3h2 d3d4 e5d4 c3d4
info depth 16 nodes 800 nps 1000 score cp 440 winrate 66.32% time 790 pv a7g1 g2g1 h3h2 c3c4 a8b8 d3d4 e5d4
info depth 16 nodes 800 nps 2758 score cp -538 winrate 30.40% time 280 pv g2g1 h3h2 d1e1 e8f8 d3d4 e5d4 c3d4
info depth 16 nodes 800 nps 1777 score cp 519 winrate 69.00% time 440 pv h3h2 c3c4 f6h4 g1g2 h2g2 c2c3 g2h2
info depth 16 nodes 800 nps 1111 score cp -545 winrate 30.18% time 710 pv c3c4 e8f8 c4c5 h2h4 d3d4 e5d4
info depth 16 nodes 800 nps 888 score cp 559 winrate 70.29% time 890 pv e8f7 c2c3 f6h4 g1g2 h2g2 d3d4 e5d4
info depth 16 nodes 800 nps 888 score cp -547 winrate 30.12% time 890 pv c4c5 f6h4 g1h1 h2h1 f1h1 h4f6 h1g1 f7e6 d1e2 e7c8
info depth 16 nodes 800 nps 987 score cp 511 winrate 68.73% time 800 pv f6g7 c2c3 g7f6 c3c4 f6h4 g1g2
info depth 16 nodes 800 nps 869 score cp -491 winrate 31.95% time 910 pv f3f4 e5f4 f1f4 f7g8 f4f3 g8h7 g1g2 h2g2
info depth 16 nodes 800 nps 888 score cp 374 winrate 64.01% time 890 pv e5f4 f1f4 f7g8 f4f3 g8h7 g1g7 h7g7 f3f7
info depth 16 nodes 800 nps 1290 score cp -398 winrate 35.13% time 610 pv f1f4 f7g8 f4g4 h2h7 d1e2 g8f7 g4g7 f7f8 g7h7
info depth 16 nodes 800 nps 4000 score cp 340 winrate 62.80% time 190 pv f7g8 f4g4 h2h7 g4g2 h7h5 g2g4 h5h2 g4g7
info depth 16 nodes 800 nps 3076 score cp -380 winrate 35.76% time 250 pv f4g4 h2h7 g4g2 h7h5 g2g4 h5h2 g4g7 g8f8
info depth 16 nodes 800 nps 1403 score cp 348 winrate 63.10% time 560 pv h2h7 g4g2 a8e8 d1c1 e7f5 e4f5
info depth 16 nodes 800 nps 3333 score cp -428 winrate 34.10% time 230 pv g4g2 a8e8 d1c1 e7f5 e4f5
info depth 16 nodes 800 nps 1111 score cp 415 winrate 65.45% time 710 pv g8h8 d1e2 g7c3 g1f1 c3b2
info depth 16 nodes 800 nps 1311 score cp -491 winrate 31.96% time 600 pv g1f1 g7c3 f1f3 c3b2 f3f1 b2c3
info depth 16 nodes 800 nps 1000 score cp 425 winrate 65.79% time 790 pv e7g8 f1f3 g7a1 f3f5 h7g7
info depth 16 nodes 800 nps 898 score cp -503 winrate 31.56% time 880 pv f1f3 g7f8 f3f5 f8g7 g2g7 h7g7
info depth 16 nodes 800 nps 860 score cp 533 winrate 69.45% time 920 pv a8f8 f3g3 g7c3 g3f3 f8f3
info depth 16 nodes 800 nps 1159 score cp -602 winrate 28.36% time 680 pv d1e2 f8f3 e2f3 g7b2 g2g8 h8g8 f3e3 g8f7 d3d4
info depth 16 nodes 800 nps 1025 score cp 545 winrate 69.82% time 770 pv f8f3 e2f3 g7d4 g2g5 d4c5 g5c5 b7b6 c5c6 g8e7
info depth 16 nodes 800 nps 2500 score cp -615 winrate 27.95% time 310 pv e2f3 g7f8 f3f2 f8c5 f2e2 c5b6 g2g8 h8g8 e4e5
info depth 16 nodes 800 nps 13333 score cp 600 winrate 71.58% time 50 pv g7f8 f3f2 f8c5 f2e2 c5b6 g2g8 h8g8 e4e5
info depth 16 nodes 800 nps 1066 score cp -600 winrate 28.41% time 740 pv d3d4 g8f6 e4e5 f6d5 f3e4 d5b4 e5e6 b4c2
info depth 16 nodes 800 nps 1176 score cp 519 winrate 68.98% time 670 pv f8e7 g2d2 h7h5 f3f4 e7f6
info depth 16 nodes 800 nps 842 score cp -488 winrate 32.05% time 940 pv g2d2 g8h6 d4d5 e7c5 d5c6 b7c6 c2c3
info depth 16 nodes 800 nps 1025 score cp 447 winrate 66.58% time 770 pv e7g5 d2d1 g5f6 c2c3 f6d4 c3d4
info depth 16 nodes 800 nps 941 score cp -486 winrate 32.10% time 840 pv d2d1 g5e7 d1b1 e7c5 d4c5 h7h5
info depth 16 nodes 800 nps 1632 score cp 394 winrate 64.74% time 480 pv g5f6 e4e5 f6e5 d4e5 g8e7 d1d6 e7f5
info depth 16 nodes 800 nps 869 score cp -366 winrate 36.26% time 910 pv e4e5 f6e7 d1b1 h7h3 f3g4 h3h2
info depth 16 nodes 800 nps 1481 score cp 290 winrate 60.98% time 530 pv f6e7 d1b1 h7h3 f3g2 h3h5 b1b7
info depth 16 nodes 800 nps 1025 score cp -290 winrate 38.99% time 770 pv d1b1 e7d8 e5e6 g8e7 b1b7 e7c8 b7h7 h8h7
info depth 16 nodes 800 nps 1739 score cp 232 winrate 58.84% time 450 pv e7d8 e5e6 h7h3 f3g4 h3h2 b1b7
info depth 16 nodes 800 nps 952 score cp -223 winrate 41.47% time 830 pv f3e4 d8e7 b1b7 e7c5 d4c5 h7b7 e5e6 b7e7
info depth 16 nodes 800 nps 941 score cp 160 winrate 56.13% time 840 pv d8e7 e4f5 e7d8 e5e6 g8e7 f5e5 e7d5 b1b7
info depth 16 nodes 800 nps 851 score cp 82 winrate 53.16% time 930 pv b1b7 h7h4 e4e3 h4h2 b7e7 g8e7 e5e6 e7d5
info depth 16 nodes 800 nps 2962 score cp -148 winrate 44.30% time 260 pv h7h4 e4e3 h4h2 b7e7 g8e7 e5e6 e7d5
info depth 16 nodes 800 nps 1269 score cp 106 winrate 54.10% time 620 pv e4d3 h4h3 d3c4 e7g5 e5e6 g5f6 d4d5
info depth 16 nodes 800 nps 1818 score cp -181 winrate 43.05% time 430 pv h4h3 d3c4 e7g5 e5e6 g5d8 d4d5 c6d5
info depth 16 nodes 800 nps 1666 score cp 143 winrate 55.49% time 470 pv d3c4 e7d8 b7b8 h3h6 b8d8 h8h7
info depth 16 nodes 800 nps 1600 score cp -219 winrate 41.65% time 490 pv h3e3 b7b8 h8g7 b8a8 e7d8 a8d8
info depth 16 nodes 800 nps 1012 score cp 278 winrate 60.57% time 780 pv b7c7 e7d8 c7c6 e3h3 c6d6 h3h1 d6d8
info depth 16 nodes 800 nps 1333 score cp -362 winrate 36.43% time 590 pv e7d8 c7c6 g8e7 c6d6 h8g7 d6d8 g7f7 d8d6
info depth 16 nodes 800 nps 1600 score cp 357 winrate 63.43% time 490 pv c7c6 e3e1 d4d5 e1e5 d5d6 e5c5 c4c5 h8h7
info depth 16 nodes 800 nps 6666 score cp -410 winrate 34.72% time 110 pv e3e1 d4d5 e1e5 d5d6 e5c5 c4c5 h8h7
info depth 16 nodes 800 nps 963 score cp 366 winrate 63.73% time 820 pv c4b5 h8h7 c6d6 d8c7 e5e6 c7d6
info depth 16 nodes 800 nps 1159 score cp -467 winrate 32.77% time 680 pv g8e7 c6d6 d8c7 d6d7 c7b8 d7e7 h8g8
info depth 16 nodes 800 nps 888 score cp 597 winrate 71.50% time 890 pv c6d6 d8c7 d6d7 c7e5 d4e5 e1e4 d7e7 e4a4
info depth 16 nodes 800 nps 4444 score cp -661 winrate 26.56% time 170 pv d8c7 d6d7 e1b1 b5c4 c7b8 d7e7 h8g8 c5c6
info depth 16 nodes 800 nps 1509 score cp 648 winrate 73.05% time 520 pv d6d7 e1b1 b5c4 b1b4 c4c3 b4a4 d7c7 a4a3 c3b2
info depth 16 nodes 800 nps 2051 score cp -763 winrate 23.61% time 380 pv e1b1 b5c4 b1b4 c4c3 b4a4 d7c7 a4a3 c3b2
info depth 16 nodes 800 nps 1012 score cp 716 winrate 75.07% time 780 pv b5c4 b1c1 c4b5 c1c2 d7e7 h8g8 e7c7 g8f8 b5a5
info depth 16 nodes 800 nps 1818 score cp -705 winrate 25.25% time 430 pv b1c1 c4b5 e7c8 e5e6 c1c2 e6e7 c8e7 d7e7 h8g8
info depth 15 nodes 800 nps 1012 score cp 894 winrate 79.85% time 780 pv d7c7 c1c2 c4b5 e7d5 c7a7 d5c3 b5a5
info depth 15 nodes 800 nps 2352 score cp -1062 winrate 16.32% time 330 pv c1c2 c4b5 e7d5 c7a7 d5c3 b5a5
info depth 15 nodes 800 nps 1702 score cp 1031 winrate 83.01% time 460 pv c4d3 c2a2 c7e7 a2a4 d4d5 h8g8 d5d6 g8f8
info depth 15 nodes 800 nps 1290 score cp -1098 winrate 15.58% time 610 pv c2a2 c7e7 a2a4 c5c6 h8g8 c6c7 a4d4 d3d4
info depth 15 nodes 800 nps 1290 score cp 1033 winrate 83.07% time 610 pv c7e7 a2a4 c5c6 a4a3 d3c4 a5a4 c6c7 a3a2
info depth 15 nodes 800 nps 2962 score cp -1138 winrate 14.78% time 260 pv a2a4 c5c6 h8g8 c6c7 g8f8 c7c8q f8e7 c8b7 e7e6
info depth 15 nodes 800 nps 1403 score cp 1189 winrate 86.17% time 560 pv c5c6 h8g8 c6c7 g8f8 c7c8q f8e7 c8b7 e7e6 d4d5
info depth 15 nodes 800 nps 1860 score cp -1300 winrate 11.91% time 420 pv h8g8 c6c7 g8f8 c7c8q f8e7 c8c7 e7e6 d3c3 e6d5
info depth 15 nodes 800 nps 1538 score cp 1403 winrate 89.66% time 510 pv c6c7 g8f8 e7d7 a4a3 d3c4 a3a2 c7c8q
info depth 15 nodes 800 nps 3809 score cp -1538 winrate  8.58% time 200 pv a4a3 d3c4 g8f8 e7d7 f8e8 c7c8q
info depth 15 nodes 800 nps 2352 score cp 1535 winrate 91.39% time 330 pv d3c4 g8f8 e7d7 f8e8 c7c8q
info depth 14 nodes 800 nps 5333 score cp -1710 winrate  6.72% time 140 pv g8f8 e7d7 f8e8 c7c8q
info depth 14 nodes 800 nps 4705 score cp 1757 winrate 93.73% time 160 pv e7d7 f8e8 c7c8q
info depth 13 nodes 800 nps 13333 score cp -2216 winrate  3.20% time 50 pv f8e8 c7c8q
info depth 11 nodes 800 nps 10000 score cp 2597 winrate 98.19% time 70 pv c7c8q
PGN
1. d3 a5 2. Nf3 Nf6 3. Nc3 Ng4 4. h3 Nf6 5. a4 Nc6 6. Rb1 d5 7. Bg5 h6 8.
Bh4 d4 9. e4 dxc3 10. bxc3 e5 11. g4 g5 12. Bg3 Nxg4 13. hxg4 Bxg4 14. Be2
f6 15. Nxg5 Bxe2 16. Qxe2 fxg5 17. f3 h5 18. Qg2 Qb8 19. Ke2 h4 20. Be1 Bh6
21. Bxh4 gxh4 22. Rxh4 Bg7 23. Rg4 Bf6 24. Qg1 Ne7 25. Rg2 Rh3 26. Rf1 c6
27. Ke1 Qa7 28. Kd1 Qxg1 29. Rgxg1 Rh2 30. c4 Kf7 31. c5 Bg7 32. f4 exf4
33. Rxf4+ Kg8 34. Rfg4 Rh7 35. R4g2 Kh8 36. Rf1 Ng8 37. Rf3 Rf8 38. Ke2
Rxf3 39. Kxf3 Bf8 40. d4 Be7 41. Rd2 Bg5 42. Rd1 Bf6 43. e5 Be7 44. Rb1 Bd8
45. Ke4 Be7 46. Rxb7 Rh4+ 47. Kd3 Rh3+ 48. Kc4 Re3 49. Rc7 Bd8 50. Rxc6 Re1
51. Kb5 Ne7 52. Rd6 Bc7 53. Rd7 Rb1+ 54. Kc4 Rc1 55. Rxc7 Rxc2+ 56. Kd3 Ra2
57. Rxe7 Rxa4 58. c6 Kg8 59. c7 Ra3+ 60. Kc4 Kf8 61. Rd7 Ke8 62. c8=Q+
END
Score: 1

F:\Chess\LCZero\test>

[jjoshua2]

1. d3 a5 2. Nf3 Nf6 3. Nc3 Ng4 4. h3 Nf6 5. a4 Nc6 6. Rb1 d5 7. Bg5 h6 8.
Bh4 d4 9. e4 dxc3 10. bxc3 e5 11. g4 g5 12. Bg3 Nxg4 13. hxg4 Bxg4 14. Be2
f6 15. Nxg5 Bxe2 16. Qxe2 fxg5 17. f3 h5 18. Qg2 Qb8 19. Ke2 h4 20. Be1 Bh6
21. Bxh4 gxh4 22. Rxh4 Bg7 23. Rg4 Bf6 24. Qg1 Ne7 25. Rg2 Rh3 26. Rf1 c6
27. Ke1 Qa7 28. Kd1 Qxg1 29. Rgxg1 Rh2 30. c4 Kf7 31. c5 Bg7 32. f4 exf4
33. Rxf4+ Kg8 34. Rfg4 Rh7 35. R4g2 Kh8 36. Rf1 Ng8 37. Rf3 Rf8 38. Ke2
Rxf3 39. Kxf3 Bf8 40. d4 Be7 41. Rd2 Bg5 42. Rd1 Bf6 43. e5 Be7 44. Rb1 Bd8
45. Ke4 Be7 46. Rxb7 Rh4+ 47. Kd3 Rh3+ 48. Kc4 Re3 49. Rc7 Bd8 50. Rxc6 Re1
51. Kb5 Ne7 52. Rd6 Bc7 53. Rd7 Rb1+ 54. Kc4 Rc1 55. Rxc7 Rxc2+ 56. Kd3 Ra2
57. Rxe7 Rxa4 58. c6 Kg8 59. c7 Ra3+ 60. Kc4 Kf8 61. Rd7 Ke8 62. c8=Q+
END
Score: 1

[CMCanavessi]

Exactly the same game

[jkiliani]

Looks like determinism with a given seed is actually working... surprising given the OpenCL errors...
Oh right it was CPU only, got it now.

[jjoshua2]

I've never had an opencl error that I've seen on windows x64 with either intel integrated gpu or NVidia 1080 ti. Reproducibility is good. Would let us use boinc with replay to catch cheaters.
My match game Gen9 50k playouts just lost to SF9 1 ply.

[Error323]

Bah, the games indeed are equivalent and I get the exact same winrates as @CMCanavessi, but the binary files differ from mine _and_ they differ from eachother :( @evalon32 do you have an idea why?

This is the script I use for testing:

#!/usr/bin/env python3

import sys
import chunkparser
import numpy as np
import gzip

with gzip.open(sys.argv[1], 'rb') as f:
    v1 = f.read()

with gzip.open(sys.argv[2], 'rb') as f:
    v2 = f.read()

parser = chunkparser.ChunkParser(chunkparser.ChunkDataSrc([v1, v2]))
gen1 = parser.convert_chunkdata_to_v2(v1)
gen2 = parser.convert_chunkdata_to_v2(v2)

for t1 in gen1:
    t2 = next(gen2)
    t1 = parser.convert_v2_to_tuple(t1)
    t2 = parser.convert_v2_to_tuple(t2)
    assert(t1[0] == t2[0])
    assert(t1[1] == t2[1])
    assert(t1[2] == t2[2])

# drain the parser
for _ in parser.parse():
    pass

[Error323]

@CMCanavessi , @jjoshua2 could you now use your current binary and run the program with the same arguments again? Then upload the results, make sure the PGN is equal.

[jjoshua2]

which binary? that branch, or latest master? didn't we just do this?

[Error323]

Latest master!

[jjoshua2]

It doesn't have an exe yet https://ci.appveyor.com/project/glinscott/leela-chess/build/job/pj54uy4s4l4uwptu/artifacts

[CMCanavessi]

I'll get the new one when it's ready

[Error323]

The one you are using yourself I mean. Not the one from the branch #119.

[jjoshua2]

the Fix assert for bit planes exe is out so ill use that

[Error323]

no no, that won't help. Please use your main lczero.exe that you're using for training. Also, do not use the binary from #119 for training and uploading to the server. You will corrupt the training data if you do. Which stalls learning.

[jjoshua2]

ok, I did a couple games, but I stopped it. training.0.gz

PS C:\Users\Mark\Downloads\leela-chess - Copy (3)> .\lczero.exe -w .\networks\5c8d14d5b1acb2f34f422e152465e005987328af7802939a131ac3f32c1f3215 --start bench^C
PS C:\Users\Mark\Downloads\leela-chess - Copy (3)> ./lczero --weights=gen8 --seed=1 --start='train 1 1' -t1 -q
info depth 16 nodes 800 nps 1025 score cp 15 winrate 50.61% time 770 pv d2d3 g8f6 g1f3 d7d6 e2e4
info depth 16 nodes 800 nps 1250 score cp -3 winrate 49.86% time 630 pv a7a5 g1f3 g8f6 b1c3 f6g4
info depth 16 nodes 800 nps 1176 score cp -6 winrate 49.76% time 670 pv g1f3 g8f6 b1a3 f6g4
info depth 16 nodes 800 nps 1111 score cp -11 winrate 49.56% time 710 pv g8f6 a2a4 d7d6 b1c3
info depth 17 nodes 800 nps 1081 score cp -13 winrate 49.47% time 730 pv b1c3 d7d6 a2a4 f6g4
info depth 17 nodes 800 nps 1126 score cp 4 winrate 50.18% time 700 pv f6g4 e2e4 g4h2 f3h2 d7d6
info depth 17 nodes 800 nps 1111 score cp 22 winrate 50.88% time 710 pv h2h3 d7d6 h3g4 c8g4 h1g1 g4f3 e2f3
info depth 17 nodes 800 nps 1176 score cp -28 winrate 48.91% time 670 pv g4f6 a2a4 b7b6 f3g5
info depth 17 nodes 800 nps 1095 score cp 6 winrate 50.25% time 720 pv a2a4 b8c6 c3b5 c6b4
info depth 17 nodes 800 nps 1355 score cp -29 winrate 48.87% time 580 pv b8c6 e2e4 c6b4 e4e5
info depth 17 nodes 800 nps 1428 score cp 10 winrate 50.41% time 550 pv a1b1 c6b4 e2e4 f6e4 d3e4
info depth 16 nodes 800 nps 1311 score cp -55 winrate 47.86% time 600 pv d7d5 c1d2 d5d4 c3b5
info depth 17 nodes 800 nps 1568 score cp 62 winrate 52.39% time 500 pv c1g5 h7h6 g5f6 g7f6 e2e4
info depth 17 nodes 800 nps 1428 score cp -39 winrate 48.48% time 550 pv h7h6 g5f4 d5d4 c3b5 c6b4
info depth 17 nodes 800 nps 1428 score cp -1 winrate 49.93% time 550 pv g5h4 c8d7 e2e3 g7g5 h4g5
info depth 17 nodes 800 nps 1454 score cp -23 winrate 49.10% time 540 pv d5d4 e2e4 d4c3 b2c3 b7b6 e4e5
info depth 17 nodes 800 nps 1290 score cp 0 winrate 50.04% time 610 pv e2e4 d4c3 b2c3 b7b6 e4e5 c8b7
info depth 17 nodes 800 nps 1311 score cp -7 winrate 49.72% time 600 pv d4c3 b2c3 c8h3 g2h3 f6e4 d3e4 e7e6
info depth 17 nodes 800 nps 1509 score cp -23 winrate 49.09% time 520 pv b2c3 g7g5 f3g5 h6g5 h4g5 h8g8 g5f6 e7f6
info depth 17 nodes 800 nps 6153 score cp -2 winrate 49.89% time 120 pv e7e5 h4g3 h6h5 f3e5
info depth 17 nodes 800 nps 1250 score cp -22 winrate 49.15% time 630 pv g2g4 g7g5 h4g5 h6g5 f3g5 c8g4 h3g4
info depth 17 nodes 800 nps 1230 score cp 14 winrate 50.56% time 640 pv g7g5 f3g5 h6g5 h4g5 c8g4 g5f6 g4d1 f6d8
info depth 17 nodes 800 nps 1428 score cp -125 winrate 45.20% time 550 pv h4g3 c8d7 f3e5 c6e5 g3e5 d7a4 e5f6 d8f6
info depth 17 nodes 800 nps 1568 score cp 79 winrate 53.05% time 500 pv f6g4 h3g4 c8g4 g3e5 c6e5 f3e5
info depth 17 nodes 800 nps 1250 score cp -78 winrate 46.97% time 630 pv h3g4 c8g4 h1g1 g4f3 d1f3 g5g4 f3g4 f8e7
info depth 17 nodes 800 nps 1403 score cp 107 winrate 54.13% time 560 pv c8g4 b1b7 g4f3 d1f3 f8e7 f3d1 g5g4 g3e5
info depth 17 nodes 800 nps 4000 score cp -155 winrate 44.06% time 190 pv f1e2 h8g8 f3e5 c6e5 g3e5 g4e2
info depth 17 nodes 800 nps 1355 score cp 98 winrate 53.78% time 580 pv f7f6 f3e5 c6e5 e2g4 e5g4 d1g4 c7c6
info depth 17 nodes 800 nps 1269 score cp -44 winrate 48.29% time 620 pv f3g5 g4e2 d1e2 f6g5 e2h5 e8e7 g3e5 c6e5
info depth 17 nodes 800 nps 2105 score cp 49 winrate 51.89% time 370 pv g4e2 d1e2 f6g5 f2f3 h8g8 g3e5 c6e5 b1d1
info depth 17 nodes 800 nps 3076 score cp -130 winrate 45.01% time 250 pv d1e2 f6g5 h1g1 h8g8 g3e5 c6e5 b1d1 f8c5
info depth 17 nodes 800 nps 2962 score cp 89 winrate 53.44% time 260 pv f6g5 f2f3 h8g8 b1b7 f8e7 g3e5
info depth 17 nodes 800 nps 3200 score cp -154 winrate 44.08% time 240 pv f2f3 h8g8 b1b7 f8e7 g3e5 c6e5
info depth 17 nodes 800 nps 1081 score cp 155 winrate 55.94% time 730 pv h6h5 b1b2 h8g8 g3e5 c6e5
info depth 17 nodes 800 nps 1159 score cp -148 winrate 44.32% time 680 pv e2g2 h5h4 g3h4 g5h4 b1d1 h4h3 g2h3
info depth 17 nodes 800 nps 1142 score cp 120 winrate 54.60% time 690 pv d8b8 h1g1 g5g4 f3g4
info depth 17 nodes 800 nps 1081 score cp -103 winrate 46.05% time 730 pv e1e2 f8e7 g3e5 c6e5 d3d4
info depth 17 nodes 800 nps 1111 score cp 184 winrate 57.03% time 710 pv h5h4 e2e3 h4h3 g2h3 h8h3 h1h3
info depth 17 nodes 800 nps 1311 score cp -155 winrate 44.06% time 600 pv g3e1 f8h6 e1h4 g5h4 h1h4 h8g8 h4h6
info depth 17 nodes 800 nps 1403 score cp 117 winrate 54.52% time 560 pv f8h6 e1h4 g5h4 h1h4 h6g7 h4h1 h8h1
info depth 17 nodes 800 nps 1403 score cp -134 winrate 44.83% time 560 pv e1h4 g5h4 h1h4 h8g8 h4h6 g8g2 e2e3
info depth 17 nodes 800 nps 1142 score cp 169 winrate 56.48% time 690 pv g5h4 h1h4 b8a7 h4h6 h8h6 d3d4 e5d4 c3d4
info depth 17 nodes 800 nps 5714 score cp -208 winrate 42.07% time 130 pv h1h4 b8a7 h4h6 h8h6 d3d4 e5d4 c3d4
info depth 17 nodes 800 nps 1666 score cp 189 winrate 57.24% time 470 pv h6g7 h4h8 g7h8 g2g6 e8e7 b1b2 c6d8
info depth 17 nodes 800 nps 1860 score cp -192 winrate 42.65% time 420 pv h4g4 g7f6 g2g3 h8g8 g4g8 e8d7
info depth 17 nodes 800 nps 1568 score cp 122 winrate 54.69% time 500 pv g7f6 g2g1 b8d8 g1g3 h8f8 g3h4
info depth 17 nodes 800 nps 2424 score cp -188 winrate 42.79% time 320 pv g2g1 b8d8 g1g3 f6e7 g4g7
info depth 17 nodes 800 nps 1311 score cp 230 winrate 58.78% time 600 pv c6e7 g4g2 c7c6 b1b2 h8f8 b2b1
info depth 17 nodes 800 nps 2000 score cp -293 winrate 38.91% time 390 pv g4g2 c7c6 b1b2 h8f8 b2b1
info depth 17 nodes 800 nps 1777 score cp 264 winrate 60.05% time 440 pv h8h3 b1b2 c7c6 g1f2 b8c7 f2g1
info depth 17 nodes 800 nps 1568 score cp -301 winrate 38.59% time 500 pv b1f1 c7c6 g1f2 b8c7 f2g1 h3h4
info depth 17 nodes 800 nps 1269 score cp 239 winrate 59.11% time 620 pv c7c6 g1f2 b8c7 f2g1 h3h4 g2g3
info depth 17 nodes 800 nps 1250 score cp -308 winrate 38.37% time 630 pv e2e1 h3h4 e1f2 h4h2 g1h2
info depth 17 nodes 800 nps 1290 score cp 326 winrate 62.29% time 610 pv b8a7 e1e2 a7g1 f1g1 h3h4 g2g3 h4h2 e2d1 f6h4
info depth 16 nodes 800 nps 2424 score cp -388 winrate 35.47% time 320 pv e1d1 a7g1 g2g1 h3h2 d3d4 e5d4 c3d4
info depth 16 nodes 800 nps 1538 score cp 440 winrate 66.32% time 510 pv a7g1 g2g1 h3h2 c3c4 a8b8 d3d4 e5d4
info depth 16 nodes 800 nps 4000 score cp -538 winrate 30.40% time 190 pv g2g1 h3h2 d1e1 e8f8 d3d4 e5d4 c3d4
info depth 16 nodes 800 nps 2666 score cp 519 winrate 69.00% time 290 pv h3h2 c3c4 f6h4 g1g2 h2g2 c2c3 g2h2
info depth 16 nodes 800 nps 1666 score cp -545 winrate 30.18% time 470 pv c3c4 e8f8 c4c5 h2h4 d3d4 e5d4
info depth 16 nodes 800 nps 1311 score cp 559 winrate 70.29% time 600 pv e8f7 c2c3 f6h4 g1g2 h2g2 d3d4 e5d4
info depth 16 nodes 800 nps 1428 score cp -547 winrate 30.12% time 550 pv c4c5 f6h4 g1h1 h2h1 f1h1 h4f6 h1g1 f7e6 d1e2 e7c8
info depth 16 nodes 800 nps 1509 score cp 511 winrate 68.73% time 520 pv f6g7 c2c3 g7f6 c3c4 f6h4 g1g2
info depth 16 nodes 800 nps 1454 score cp -491 winrate 31.95% time 540 pv f3f4 e5f4 f1f4 f7g8 f4f3 g8h7 g1g2 h2g2
info depth 16 nodes 800 nps 1481 score cp 374 winrate 64.01% time 530 pv e5f4 f1f4 f7g8 f4f3 g8h7 g1g7 h7g7 f3f7
info depth 16 nodes 800 nps 1860 score cp -398 winrate 35.13% time 420 pv f1f4 f7g8 f4g4 h2h7 d1e2 g8f7 g4g7 f7f8 g7h7
info depth 16 nodes 800 nps 5714 score cp 340 winrate 62.80% time 130 pv f7g8 f4g4 h2h7 g4g2 h7h5 g2g4 h5h2 g4g7
info depth 16 nodes 800 nps 4705 score cp -380 winrate 35.76% time 160 pv f4g4 h2h7 g4g2 h7h5 g2g4 h5h2 g4g7 g8f8
info depth 16 nodes 800 nps 1904 score cp 348 winrate 63.10% time 410 pv h2h7 g4g2 a8e8 d1c1 e7f5 e4f5
info depth 16 nodes 800 nps 4444 score cp -428 winrate 34.10% time 170 pv g4g2 a8e8 d1c1 e7f5 e4f5
info depth 16 nodes 800 nps 1739 score cp 415 winrate 65.45% time 450 pv g8h8 d1e2 g7c3 g1f1 c3b2
info depth 16 nodes 800 nps 2162 score cp -491 winrate 31.96% time 360 pv g1f1 g7c3 f1f3 c3b2 f3f1 b2c3
info depth 16 nodes 800 nps 1568 score cp 425 winrate 65.79% time 500 pv e7g8 f1f3 g7a1 f3f5 h7g7
info depth 16 nodes 800 nps 1290 score cp -503 winrate 31.56% time 610 pv f1f3 g7f8 f3f5 f8g7 g2g7 h7g7
info depth 16 nodes 800 nps 1355 score cp 533 winrate 69.45% time 580 pv a8f8 f3g3 g7c3 g3f3 f8f3
info depth 16 nodes 800 nps 1777 score cp -602 winrate 28.36% time 440 pv d1e2 f8f3 e2f3 g7b2 g2g8 h8g8 f3e3 g8f7 d3d4
info depth 16 nodes 800 nps 1481 score cp 545 winrate 69.82% time 530 pv f8f3 e2f3 g7d4 g2g5 d4c5 g5c5 b7b6 c5c6 g8e7
info depth 16 nodes 800 nps 3478 score cp -616 winrate 27.93% time 220 pv e2f3 g7f8 f3f2 f8c5 f2e2 c5b6 g2g8 h8g8 e4e5
info depth 16 nodes 800 nps 16000 score cp 600 winrate 71.57% time 40 pv g7f8 f3f2 f8c5 f2e2 c5b6 g2g8 h8g8 e4e5
info depth 16 nodes 800 nps 1538 score cp -600 winrate 28.41% time 510 pv d3d4 g8f6 e4e5 f6d5 f3e4 d5b4 e5e6 b4c2
info depth 16 nodes 800 nps 1632 score cp 519 winrate 68.98% time 480 pv f8e7 g2d2 h7h5 f3f4 e7f6
info depth 16 nodes 800 nps 1250 score cp -488 winrate 32.05% time 630 pv g2d2 g8h6 d4d5 e7c5 d5c6 b7c6 c2c3
info depth 16 nodes 800 nps 1509 score cp 447 winrate 66.58% time 520 pv e7g5 d2d1 g5f6 c2c3 f6d4 c3d4
info depth 16 nodes 800 nps 1379 score cp -486 winrate 32.10% time 570 pv d2d1 g5e7 d1b1 e7c5 d4c5 h7h5
info depth 16 nodes 800 nps 2352 score cp 394 winrate 64.74% time 330 pv g5f6 e4e5 f6e5 d4e5 g8e7 d1d6 e7f5
info depth 16 nodes 800 nps 1269 score cp -366 winrate 36.26% time 620 pv e4e5 f6e7 d1b1 h7h3 f3g4 h3h2
info depth 16 nodes 800 nps 2051 score cp 290 winrate 60.98% time 380 pv f6e7 d1b1 h7h3 f3g2 h3h5 b1b7
info depth 16 nodes 800 nps 1509 score cp -290 winrate 38.99% time 520 pv d1b1 e7d8 e5e6 g8e7 b1b7 e7c8 b7h7 h8h7
info depth 16 nodes 800 nps 2857 score cp 232 winrate 58.84% time 270 pv e7d8 e5e6 h7h3 f3g4 h3h2 b1b7
info depth 16 nodes 800 nps 1568 score cp -223 winrate 41.47% time 500 pv f3e4 d8e7 b1b7 e7c5 d4c5 h7b7 e5e6 b7e7
info depth 16 nodes 800 nps 1428 score cp 160 winrate 56.13% time 550 pv d8e7 e4f5 e7d8 e5e6 g8e7 f5e5 e7d5 b1b7
info depth 16 nodes 800 nps 1290 score cp 82 winrate 53.16% time 610 pv b1b7 h7h4 e4e3 h4h2 b7e7 g8e7 e5e6 e7d5
info depth 16 nodes 800 nps 4444 score cp -148 winrate 44.30% time 170 pv h7h4 e4e3 h4h2 b7e7 g8e7 e5e6 e7d5
info depth 16 nodes 800 nps 1904 score cp 106 winrate 54.10% time 410 pv e4d3 h4h3 d3c4 e7g5 e5e6 g5f6 d4d5
info depth 16 nodes 800 nps 2580 score cp -181 winrate 43.05% time 300 pv h4h3 d3c4 e7g5 e5e6 g5d8 d4d5 c6d5
info depth 16 nodes 800 nps 2285 score cp 143 winrate 55.49% time 340 pv d3c4 e7d8 b7b8 h3h6 b8d8 h8h7
info depth 16 nodes 800 nps 2162 score cp -219 winrate 41.65% time 360 pv h3e3 b7b8 h8g7 b8a8 e7d8 a8d8
info depth 16 nodes 800 nps 1454 score cp 278 winrate 60.57% time 540 pv b7c7 e7d8 c7c6 e3h3 c6d6 h3h1 d6d8
info depth 16 nodes 800 nps 1951 score cp -362 winrate 36.43% time 400 pv e7d8 c7c6 g8e7 c6d6 h8g7 d6d8 g7f7 d8d6
info depth 16 nodes 800 nps 2285 score cp 357 winrate 63.43% time 340 pv c7c6 e3e1 d4d5 e1e5 d5d6 e5c5 c4c5 h8h7
info depth 16 nodes 800 nps 8888 score cp -410 winrate 34.72% time 80 pv e3e1 d4d5 e1e5 d5d6 e5c5 c4c5 h8h7
info depth 16 nodes 800 nps 1379 score cp 366 winrate 63.73% time 570 pv c4b5 h8h7 c6d6 d8c7 e5e6 c7d6
info depth 16 nodes 800 nps 1702 score cp -467 winrate 32.77% time 460 pv g8e7 c6d6 d8c7 d6d7 c7b8 d7e7 h8g8
info depth 16 nodes 800 nps 1250 score cp 597 winrate 71.50% time 630 pv c6d6 d8c7 d6d7 c7e5 d4e5 e1e4 d7e7 e4a4
info depth 16 nodes 800 nps 6153 score cp -661 winrate 26.56% time 120 pv d8c7 d6d7 e1b1 b5c4 c7b8 d7e7 h8g8 c5c6
info depth 16 nodes 800 nps 2162 score cp 648 winrate 73.05% time 360 pv d6d7 e1b1 b5c4 b1b4 c4c3 b4a4 d7c7 a4a3 c3b2
info depth 16 nodes 800 nps 2962 score cp -763 winrate 23.61% time 260 pv e1b1 b5c4 b1b4 c4c3 b4a4 d7c7 a4a3 c3b2
info depth 16 nodes 800 nps 1538 score cp 716 winrate 75.07% time 510 pv b5c4 b1c1 c4b5 c1c2 d7e7 h8g8 e7c7 g8f8 b5a5
info depth 16 nodes 800 nps 2758 score cp -705 winrate 25.25% time 280 pv b1c1 c4b5 e7c8 e5e6 c1c2 e6e7 c8e7 d7e7 h8g8
info depth 15 nodes 800 nps 1568 score cp 894 winrate 79.85% time 500 pv d7c7 c1c2 c4b5 e7d5 c7a7 d5c3 b5a5
info depth 15 nodes 800 nps 3636 score cp -1062 winrate 16.32% time 210 pv c1c2 c4b5 e7d5 c7a7 d5c3 b5a5
info depth 15 nodes 800 nps 2352 score cp 1031 winrate 83.01% time 330 pv c4d3 c2a2 c7e7 a2a4 d4d5 h8g8 d5d6 g8f8
info depth 15 nodes 800 nps 1777 score cp -1098 winrate 15.58% time 440 pv c2a2 c7e7 a2a4 c5c6 h8g8 c6c7 a4d4 d3d4
info depth 15 nodes 800 nps 2000 score cp 1033 winrate 83.07% time 390 pv c7e7 a2a4 c5c6 a4a3 d3c4 a5a4 c6c7 a3a2
info depth 15 nodes 800 nps 4210 score cp -1138 winrate 14.78% time 180 pv a2a4 c5c6 h8g8 c6c7 g8f8 c7c8q f8e7 c8b7 e7e6
info depth 15 nodes 800 nps 1951 score cp 1189 winrate 86.17% time 400 pv c5c6 h8g8 c6c7 g8f8 c7c8q f8e7 c8b7 e7e6 d4d5
info depth 15 nodes 800 nps 2758 score cp -1300 winrate 11.91% time 280 pv h8g8 c6c7 g8f8 c7c8q f8e7 c8c7 e7e6 d3c3 e6d5
info depth 15 nodes 800 nps 2352 score cp 1403 winrate 89.66% time 330 pv c6c7 g8f8 e7d7 a4a3 d3c4 a3a2 c7c8q
info depth 15 nodes 800 nps 5000 score cp -1538 winrate  8.58% time 150 pv a4a3 d3c4 g8f8 e7d7 f8e8 c7c8q
info depth 15 nodes 800 nps 3478 score cp 1535 winrate 91.39% time 220 pv d3c4 g8f8 e7d7 f8e8 c7c8q
info depth 14 nodes 800 nps 8000 score cp -1710 winrate  6.72% time 90 pv g8f8 e7d7 f8e8 c7c8q
info depth 14 nodes 800 nps 7272 score cp 1757 winrate 93.73% time 100 pv e7d7 f8e8 c7c8q
info depth 13 nodes 800 nps 20000 score cp -2216 winrate  3.20% time 30 pv f8e8 c7c8q
info depth 11 nodes 800 nps 16000 score cp 2597 winrate 98.19% time 40 pv c7c8q
PGN
1. d3 a5 2. Nf3 Nf6 3. Nc3 Ng4 4. h3 Nf6 5. a4 Nc6 6. Rb1 d5 7. Bg5 h6 8.
Bh4 d4 9. e4 dxc3 10. bxc3 e5 11. g4 g5 12. Bg3 Nxg4 13. hxg4 Bxg4 14. Be2
f6 15. Nxg5 Bxe2 16. Qxe2 fxg5 17. f3 h5 18. Qg2 Qb8 19. Ke2 h4 20. Be1 Bh6
21. Bxh4 gxh4 22. Rxh4 Bg7 23. Rg4 Bf6 24. Qg1 Ne7 25. Rg2 Rh3 26. Rf1 c6
27. Ke1 Qa7 28. Kd1 Qxg1 29. Rgxg1 Rh2 30. c4 Kf7 31. c5 Bg7 32. f4 exf4
33. Rxf4+ Kg8 34. Rfg4 Rh7 35. R4g2 Kh8 36. Rf1 Ng8 37. Rf3 Rf8 38. Ke2
Rxf3 39. Kxf3 Bf8 40. d4 Be7 41. Rd2 Bg5 42. Rd1 Bf6 43. e5 Be7 44. Rb1 Bd8
45. Ke4 Be7 46. Rxb7 Rh4+ 47. Kd3 Rh3+ 48. Kc4 Re3 49. Rc7 Bd8 50. Rxc6 Re1
51. Kb5 Ne7 52. Rd6 Bc7 53. Rd7 Rb1+ 54. Kc4 Rc1 55. Rxc7 Rxc2+ 56. Kd3 Ra2
57. Rxe7 Rxa4 58. c6 Kg8 59. c7 Ra3+ 60. Kc4 Kf8 61. Rd7 Ke8 62. c8=Q+
END
Score: 1

[CMCanavessi]

I'm constantly replacing my "testing binary" with the latest version, but I have saved the one I used for Gen 4, Gen 6 and Gen 8. They are quite old already, should i still try those?

[jjoshua2]

Can you link me an exe to use? I got the latest one from here https://ci.appveyor.com/project/glinscott/leela-chess

[Error323]

Any version from the master branch is good. You should use that. What we did just now was a data integrity test. Which failed. Please trash the binaries and never use them again :sweat_smile:

[jjoshua2]

I thought my link was the master branch, and it says Fix assert for bit planes

[Error323]

@jjoshua2 https://ci.appveyor.com/project/glinscott/leela-chess/branch/master that's the master branch.

[jjoshua2]

ok that gives: training.0.gz. Seems to be the same pgn as my previous two exes

```info depth 16 nodes 800 nps 2758 score cp -705 winrate 25.25% time 280 pv b1c1 c4b5 e7c8 e5e6 c1c2 e6e7 c8e7 d7e7 h8g8
info depth 15 nodes 800 nps 1600 score cp 894 winrate 79.85% time 490 pv d7c7 c1c2 c4b5 e7d5 c7a7 d5c3 b5a5
info depth 15 nodes 800 nps 3333 score cp -1062 winrate 16.32% time 230 pv c1c2 c4b5 e7d5 c7a7 d5c3 b5a5
info depth 15 nodes 800 nps 2666 score cp 1031 winrate 83.01% time 290 pv c4d3 c2a2 c7e7 a2a4 d4d5 h8g8 d5d6 g8f8
info depth 15 nodes 800 nps 2051 score cp -1098 winrate 15.58% time 380 pv c2a2 c7e7 a2a4 c5c6 h8g8 c6c7 a4d4 d3d4
info depth 15 nodes 800 nps 1951 score cp 1033 winrate 83.07% time 400 pv c7e7 a2a4 c5c6 a4a3 d3c4 a5a4 c6c7 a3a2
info depth 15 nodes 800 nps 4210 score cp -1138 winrate 14.78% time 180 pv a2a4 c5c6 h8g8 c6c7 g8f8 c7c8q f8e7 c8b7 e7e6
info depth 15 nodes 800 nps 2000 score cp 1189 winrate 86.17% time 390 pv c5c6 h8g8 c6c7 g8f8 c7c8q f8e7 c8b7 e7e6 d4d5
info depth 15 nodes 800 nps 2758 score cp -1300 winrate 11.91% time 280 pv h8g8 c6c7 g8f8 c7c8q f8e7 c8c7 e7e6 d3c3 e6d5
info depth 15 nodes 800 nps 2424 score cp 1403 winrate 89.66% time 320 pv c6c7 g8f8 e7d7 a4a3 d3c4 a3a2 c7c8q
info depth 15 nodes 800 nps 6153 score cp -1538 winrate 8.58% time 120 pv a4a3 d3c4 g8f8 e7d7 f8e8 c7c8q
info depth 15 nodes 800 nps 4000 score cp 1535 winrate 91.39% time 190 pv d3c4 g8f8 e7d7 f8e8 c7c8q
info depth 14 nodes 800 nps 8000 score cp -1710 winrate 6.72% time 90 pv g8f8 e7d7 f8e8 c7c8q
info depth 14 nodes 800 nps 6666 score cp 1757 winrate 93.73% time 110 pv e7d7 f8e8 c7c8q
info depth 13 nodes 800 nps 16000 score cp -2216 winrate 3.20% time 40 pv f8e8 c7c8q
info depth 11 nodes 800 nps 16000 score cp 2597 winrate 98.19% time 40 pv c7c8q
PGN

1. d3 a5 2. Nf3 Nf6 3. Nc3 Ng4 4. h3 Nf6 5. a4 Nc6 6. Rb1 d5 7. Bg5 h6 8.
   Bh4 d4 9. e4 dxc3 10. bxc3 e5 11. g4 g5 12. Bg3 Nxg4 13. hxg4 Bxg4 14. Be2
   f6 15. Nxg5 Bxe2 16. Qxe2 fxg5 17. f3 h5 18. Qg2 Qb8 19. Ke2 h4 20. Be1 Bh6
2. Bxh4 gxh4 22. Rxh4 Bg7 23. Rg4 Bf6 24. Qg1 Ne7 25. Rg2 Rh3 26. Rf1 c6
3. Ke1 Qa7 28. Kd1 Qxg1 29. Rgxg1 Rh2 30. c4 Kf7 31. c5 Bg7 32. f4 exf4
4. Rxf4+ Kg8 34. Rfg4 Rh7 35. R4g2 Kh8 36. Rf1 Ng8 37. Rf3 Rf8 38. Ke2
   Rxf3 39. Kxf3 Bf8 40. d4 Be7 41. Rd2 Bg5 42. Rd1 Bf6 43. e5 Be7 44. Rb1 Bd8
5. Ke4 Be7 46. Rxb7 Rh4+ 47. Kd3 Rh3+ 48. Kc4 Re3 49. Rc7 Bd8 50. Rxc6 Re1
6. Kb5 Ne7 52. Rd6 Bc7 53. Rd7 Rb1+ 54. Kc4 Rc1 55. Rxc7 Rxc2+ 56. Kd3 Ra2
7. Rxe7 Rxa4 58. c6 Kg8 59. c7 Ra3+ 60. Kc4 Kf8 61. Rd7 Ke8 62. c8=Q+
   END
   Score: 1
   ```

[CMCanavessi]

Here are results:

Gen4 binary, dated March 09 2018 21:51
PGN
1. d3 a5 2. Nf3 Nf6 3. Nc3 Ng4 4. h3 d6 5. hxg4 Bxg4 6. Rg1 Nc6 7. Nd5 e6
8. Bg5 f6 9. Bxf6 gxf6 10. Nh2 f5 11. Ne3 h5 12. Nhxg4 fxg4 13. c3 Rg8 14.
a3 Bh6 15. Qc2 Bxe3 16. fxe3 e5 17. O-O-O h4 18. Qd2 Rb8 19. a4 g3 20. e4
h3 21. gxh3 Qh4 22. e3 Kf8 23. Qg2 Rg6 24. d4 exd4 25. exd4 Re8 26. Bb5
Rxe4 27. Rgf1+ Ke7 28. Rf3 Na7 29. Bd3 d5 30. Bf1 b6 31. Kc2 Nc6 32. Bb5
Ree6 33. Kc1 Kd7 34. c4 Rg5 35. cxd5 Rxd5 36. Rxg3 Qh5 37. Rg4 Qh6+ 38. Kc2
Kd6 39. Bxc6 Kxc6 40. Rg3 Qf4 41. Rf3 Qh4 42. Qg3 Qxg3 43. Rxg3 b5 44. Rf3
bxa4 45. h4 Kb5 46. Rd2 Rh5 47. Rf4 Re8 48. d5 Rg8 49. Kd1 Rg3 50. Re2 Rg8
51. Ree4 Kc5 52. Rxa4 Kxd5 53. Rxa5+ c5 54. b4 Rg1+ 55. Kc2 Rh6 56. Rxc5+
Kd6 57. Rc3 Rh7 58. b5 Rh1 59. b6 Rb7 60. Rb4 Kd5 61. Rc7 Rb8 62. Kd2 Kd6
63. Rh7 Kc5 64. Rb3 Rxb6 65. Rxb6 Kxb6 66. Kc2 Rh3 67. h5 Rh1 68. h6 Rh5
69. Rh8 Kc5 70. Kd3 Kd5 71. h7 Rh2 72. Ke3 Rh5 73. Kf4 Ke6 74. Kg3 Kf5 75.
Kg2 Kg4 76. Rg8+ Kf5 77. h8=Q Rxh8 78. Rxh8 Ke6 79. Kf3 Kd5 80. Rh3 Ke5 81.
Rh2 Kd4 82. Rh4+ Kd3 83. Kf4 Kd2 84. Ke4 Kc3 85. Rf4 Kd2 86. Kd4 Ke2 87.
Ke4 Kd2 88. Rf3 Ke2 89. Rh3 Kf2 90. Rh2+ Kg1 91. Rb2 Kh1 92. Ke3 Kg1 93.
Rb1+ Kg2 94. Rb2+ Kh1 95. Rb1+ Kg2 96. Rb2+ Kg1 97. Kf3 Kh1 98. Kg3 Kg1 99.
Rb1+
END
Score: 1


Gen 6 binary, dated March 13 2018 16:29
PGN
1. d3 a5 2. Nf3 Nf6 3. Nc3 Ng4 4. h3 Nf6 5. a4 Nc6 6. Rb1 d5 7. Bg5 h6 8.
Bh4 d4 9. e4 dxc3 10. bxc3 e5 11. g4 g5 12. Bg3 Nxg4 13. hxg4 Bxg4 14. Be2
f6 15. Nxg5 Bxe2 16. Qxe2 fxg5 17. f3 h5 18. Qg2 Qb8 19. Ke2 h4 20. Be1 Bh6
21. Bxh4 gxh4 22. Rxh4 Bg7 23. Rg4 Bf6 24. Qg1 Ne7 25. Rg2 Rh3 26. Rf1 c6
27. Ke1 Qa7 28. Kd1 Qxg1 29. Rgxg1 Rh2 30. c4 Kf7 31. c5 Bg7 32. f4 exf4
33. Rxf4+ Kg8 34. Rfg4 Rh7 35. R4g2 Kh8 36. Rf1 Ng8 37. Rf3 Rf8 38. Ke2
Rxf3 39. Kxf3 Bf8 40. d4 Be7 41. Rd2 Bg5 42. Rd1 Bf6 43. e5 Be7 44. Rb1 Bd8
45. Ke4 Be7 46. Rxb7 Rh4+ 47. Kd3 Rh3+ 48. Kc4 Re3 49. Rc7 Bd8 50. Rxc6 Re1
51. Kb5 Ne7 52. Rd6 Bc7 53. Rd7 Rb1+ 54. Kc4 Rc1 55. Rxc7 Rxc2+ 56. Kd3 Ra2
57. Rxe7 Rxa4 58. c6 Kg8 59. c7 Ra3+ 60. Kc4 Kf8 61. Rd7 Ke8 62. c8=Q+
END
Score: 1


Gen 8 binary, dated March 15 2018 17:08
PGN
1. d3 a5 2. Nf3 Nf6 3. Nc3 Ng4 4. h3 Nf6 5. a4 Nc6 6. Rb1 d5 7. Bg5 h6 8.
Bh4 d4 9. e4 dxc3 10. bxc3 e5 11. g4 g5 12. Bg3 Nxg4 13. hxg4 Bxg4 14. Be2
f6 15. Nxg5 Bxe2 16. Qxe2 fxg5 17. f3 h5 18. Qg2 Qb8 19. Ke2 h4 20. Be1 Bh6
21. Bxh4 gxh4 22. Rxh4 Bg7 23. Rg4 Bf6 24. Qg1 Ne7 25. Rg2 Rh3 26. Rf1 c6
27. Ke1 Qa7 28. Kd1 Qxg1 29. Rgxg1 Rh2 30. c4 Kf7 31. c5 Bg7 32. f4 exf4
33. Rxf4+ Kg8 34. Rfg4 Rh7 35. R4g2 Kh8 36. Rf1 Ng8 37. Rf3 Rf8 38. Ke2
Rxf3 39. Kxf3 Bf8 40. d4 Be7 41. Rd2 Bg5 42. Rd1 Bf6 43. e5 Be7 44. Rb1 Bd8
45. Ke4 Be7 46. Rxb7 Rh4+ 47. Kd3 Rh3+ 48. Kc4 Re3 49. Rc7 Bd8 50. Rxc6 Re1
51. Kb5 Ne7 52. Rd6 Bc7 53. Rd7 Rb1+ 54. Kc4 Rc1 55. Rxc7 Rxc2+ 56. Kd3 Ra2
57. Rxe7 Rxa4 58. c6 Kg8 59. c7 Ra3+ 60. Kc4 Kf8 61. Rd7 Ke8 62. c8=Q+
END
Score: 1


Current Master Branch binary, dated right now
PGN
1. d3 a5 2. Nf3 Nf6 3. Nc3 Ng4 4. h3 Nf6 5. a4 Nc6 6. Rb1 d5 7. Bg5 h6 8.
Bh4 d4 9. e4 dxc3 10. bxc3 e5 11. g4 g5 12. Bg3 Nxg4 13. hxg4 Bxg4 14. Be2
f6 15. Nxg5 Bxe2 16. Qxe2 fxg5 17. f3 h5 18. Qg2 Qb8 19. Ke2 h4 20. Be1 Bh6
21. Bxh4 gxh4 22. Rxh4 Bg7 23. Rg4 Bf6 24. Qg1 Ne7 25. Rg2 Rh3 26. Rf1 c6
27. Ke1 Qa7 28. Kd1 Qxg1 29. Rgxg1 Rh2 30. c4 Kf7 31. c5 Bg7 32. f4 exf4
33. Rxf4+ Kg8 34. Rfg4 Rh7 35. R4g2 Kh8 36. Rf1 Ng8 37. Rf3 Rf8 38. Ke2
Rxf3 39. Kxf3 Bf8 40. d4 Be7 41. Rd2 Bg5 42. Rd1 Bf6 43. e5 Be7 44. Rb1 Bd8
45. Ke4 Be7 46. Rxb7 Rh4+ 47. Kd3 Rh3+ 48. Kc4 Re3 49. Rc7 Bd8 50. Rxc6 Re1
51. Kb5 Ne7 52. Rd6 Bc7 53. Rd7 Rb1+ 54. Kc4 Rc1 55. Rxc7 Rxc2+ 56. Kd3 Ra2
57. Rxe7 Rxa4 58. c6 Kg8 59. c7 Ra3+ 60. Kc4 Kf8 61. Rd7 Ke8 62. c8=Q+
END
Score: 1

Damn that Gen4 binary was slow as hell without the cache improvements. Every binary produced the same, except the oldest.

[evalon32]

@Error323 No idea why, but if it helps, all diffs are in move probabilities, AFAICT. Comparing with od -A d -t f4:

4234c4234
< 0357152           0.005           0.835               0               0
---
> 0357152           0.005         0.83625               0               0
4237c4237
< 0357568               0         0.00375               0         0.02375
---
> 0357568               0         0.00375               0          0.0225
7603c7603
< 0668848           0.085               0               0          0.0525
---
> 0668848         0.08625               0               0          0.0525
7607c7607
< 0669536               0         0.00625         0.01375               0
---
> 0669536               0         0.00625          0.0125               0

[Error323]

@evalon32 This certainly helps! Nice tool! I think we are good. I think this is within the margin of error. When I have time (tonight) I'll gather some more diffs from the data by @jjoshua2 and @CMCanavessi. But this looks promising.

P.S. new net is training.

[Error323]

#!/usr/bin/env python3

import sys
import chunkparser
import numpy as np
import gzip

with gzip.open(sys.argv[1], 'rb') as f:
    v1 = f.read()

with gzip.open(sys.argv[2], 'rb') as f:
    v2 = f.read()

parser = chunkparser.ChunkParser(chunkparser.ChunkDataSrc([v1, v2]))
gen1 = parser.convert_chunkdata_to_v2(v1)
gen2 = parser.convert_chunkdata_to_v2(v2)

i = 0
s = 0
for t1 in gen1:
    t2 = next(gen2)
    t1 = parser.convert_v2_to_tuple(t1)
    t2 = parser.convert_v2_to_tuple(t2)
    p1 = np.frombuffer(t1[1], dtype=np.float32, count=1924)
    p2 = np.frombuffer(t2[1], dtype=np.float32, count=1924)
    assert(t1[0] == t2[0])
    assert(t1[2] == t2[2])
    a = np.argsort(p1[p1>0])
    b = np.argsort(p2[p2>0])
    assert((a == b).all())

# drain the parser
for _ in parser.parse():
    pass

This script sorts the legal moves on probability. All legal moves are consistent for Linux V2, Linux V1, Windows V1 and Windows V2. For both @CMCanavessi and @jjoshua2. I think we are good to go :fireworks: . The floating point probability differences are artefacts of various drivers and cards I think.

[Error323]

Training wise: The trained net looks very good in terms of accuracy and mse. But it losing to gen9 :( I'm investigating. Until we have match support Idon't want to upload poor networks.

P.S. Please update your clients, this will help training substantially!

[remdu]

This is the most common case in Leela zero. Most nets are worse than preceding one.

[jkiliani]

For Leela Zero this was a very normal thing at an early stage... at some point not every trained net is an improvement anymore. I think at this point we have to decide, do we want to follow the AlphaZero route (always promote new nets) or not? It worked for Deepmind, and the rest of the setup (800 playouts, temperature=1 for the whole game) also uses AlphaZero parameters anyway.

Personally I don't think adopting anyway will do any harm, if the training stats don't indicate there's anything wrong with the net.

[jhellis3]

Always promoting doesn't make sense AFAICS. Sure, theoretically it should continue to progress eventually, but there is certainly no guarantee of that over a given period of time. Nor is there any assurance or reason to believe that always promoting will be the most efficient path forward.

The only real benefits I can see from it is the generation of more "unique" games, which could help avoid over-fitting, and it is dead simple to do (laziest path forward). A similar effect could be accomplished by having the best network play against both itself and the latest trained network (or 2) in the event of a fail or previous best in the event of a pass. However, given how early it is and the games per day being produced, it seems likely progress is limited more by NN iterations than anything else. In this light, it is not unreasonable to just train and always promote ASAP, and maybe check every 4-5 iterations for progress. Long term, I would hope we could come up with something better....

We will also need to move to something more robust than 100 games for verification, if we want to even bother. If we always promote, there is no need to verify.

It also appears that we are currently generating far more games than we actually need per training session (at the current rate of ~1/day). Moving to 1200 or 1600 playouts would increase the quality of the games, which should benefit learning.

[jkiliani]

Raising playouts could certainly be tried, but I think it more likely that the size of the training window is going to eventually cause problems in allowing further generalisation, if that has not already happened. Leela Zero has trained on a fixed window of 250k games for a long time, here we're currently using around 100k if I understand the procedure (most recent two nets as training data) correctly. This could be too little already, although it did work in the very beginning when the networks has very little knowledge to begin with.

[jjoshua2]

I think with go someone did an experiment on 9x9 where each network was only trained on the previous 1 network and it still progressed well, so I don't think the 100k is the issue. Leela had lots of bugs early on like shuffle buffer, and it seems something like that is more likely. Maybe the annealing schedule we are using.
I also think that generating with more playouts is probably good until we have the learning system automated and working well.

[zz4032]

Can we do a test phase for 1-2 weeks? Then revert, promote only after successful matches and compare the two final versions. As far as I understood automatic matching is not ready yet anyway. And it will cost a lot of resources...

[jjoshua2]

Automatic matching not being ready does give us the perfect opportunity for this test.

[killerducky]

@jhellis3

Always promoting doesn't make sense AFAICS.

But that is what Google did so I don't think you should dismiss it so easily.

[CMCanavessi]

It can be tricky. Suppose that when you test, Gen 10 is a bit weaker than Gen 9, so you might scream REGRESSION!! and immediately reject network 10. But maybe looking closer, yes, Gen 10 is weaker but learned to castle, so there's some progress in the playing style. What would you do?
IMHO the evolving network will not always evolve in an "elo gaining" direction, but that doesn't mean it hasn't learned new tricks or different knowledge. It's complicated.

[jhellis3]

I didn't dismiss it. I simply said it doesn't make sense to me. What it does do is guarantee long term progress _eventually_, and it is dead simple to do. These two advantages combined with Google's massive resources make it a natural choice for them. However, just because it is optimal for Google short term (they wanted quick results), does not mean it is going to be the best solution in general. I am especially doubtful of its merit once the engine becomes very strong. Because every time you promote an inferior NN at that level, you are throwing away some hard earned data for the prospect of future growth. And as future growth continues to slow, the value of this proposition diminishes greatly. But until that happens it should be fine, and there will always be time to make improvements later.

[FauziAkram]

I think an important question is:How much worse is it?
if it lost 48-52 I wouldn't be so sure that its a regression overall

error323 can you please give us some numbers?

[killerducky]

What is your theory about why Google changed from match promotion in the AGZ paper to always promote in the AZ paper?

You said Google got quick results, so we should do that also? I guess you're saying do always promote at first but switch later?

[davidspencer6174]

One of the crucial insights behind the AGZ paper is that MCTS can be used as a policy improvement operator. That is, it takes a policy net P (and a value net which has to come along for the ride) and finds moves which ideally are better than those produced by P. Then the next network P' tries to approximate this improved policy function.

So the question becomes what we want from P'. Is it more important that P' wins games, or that it successfully figures out which moves P would choose with 800 playouts? I don't think the answer is at all obvious. Personally, I find the latter option more aesthetically appealing, but that doesn't necessarily mean it's correct.

I think the reason I don't like the former option as much is because it feels like a form of overfitting. Instead of taking the network that most accurately models the previous network with MCTS (the actual improvement mechanism), it takes the network which fits the end goal the best. But it is very hard to justify this rigorously, and there are certainly reasonable arguments for the former option as well.

[jjoshua2]

@killerducky I think that is reasonable. There is some a bit of extra elo to squeeze out by testing a lot of similar nets and picking the best one like leela go does. But it also seems likely that always promote could utilize resources better at the beginning during the steep climb.
I switched my clients from 800 to 1200 playouts, and they still are under a second a move.
Unrelated: is there a reason it clamps down to 2 threads even when I try more? In go 4 threads is optimal for me...

[killerducky]

@jjoshua2 you switched your own client to do something different, and it's uploading games? I don't think it's a good idea for everyone to have their own versions of the client. We should all only run the officially released client.

[jhellis3]

Google had 5000 TPUs at their disposal. And if you look at the strength curve over time, it highlights the issue quite well. Always promoting early is fine, because the NN still has much to learn and will very likely continue to progress significantly over several iterations. However, the stronger the program gets the more difficult it will be to become stronger (taking more time) especially if you are throwing away strength that took days, weeks, months to learn. But you still really want game variety (which always promoting provides), so I would instead play recent NNs vs the current best in order to get this variety. This way you never discard hard earned strength. Another option to get variety is to play the NN against itself at different strength settings, say 800 playouts vs 1600 playouts. However, I doubt this would provided as much variety as playing against other NNs, though it would be simpler to implement.

[jjoshua2]

@killerducky its all I changed. and I think a little higher quality data can only help. the learning process is supposed to be quite robust to a little bit of noise and random data
I like the idea of always promote until elo growth slows down on rating lists and then start official match testing with client and keep best and newest

[killerducky]

@jjoshua2 and what happens when @Error323 is debugging his new training code and notices some of the training files are different than the others? He could spend hours debugging that because he doesn't know 50 people out there have changed their clients to whatever they personally thought was best.

What happens when 50 people make their own personal minor change? I guarantee at least one will have bug in it. I just think this is a dangerous precedent even though this particular change is probably minor.

I think we need a central authority making these decisions.

[killerducky]

Google had 5000 TPUs at their disposal. And if you look at the strength curve over time, it highlights the issue quite well.

The Go strength curve looks similar, and they used the match promotion scheme. So I don't think this argument is valid.

[jhellis3]

You missed the point. It isn't that the method will not work, it is about efficiency over a given period of time. Every method (that makes progress) will have a similar curve on a large enough scale, simply because there is ever less improvement to be made. The point is to get as strong as possible ASAP.

[killerducky]

I agree we want to be as efficient as possible and get strong as possible ASAP. But I don't see anything in the Google papers that suggests doing matches is more efficient than always promoting. I see the opposite. They published the AGZ paper using matches. Then later they published a second paper where they always promote. I interpret this to mean always promoting is the better method. If it was less efficient why would they switch to it?

[jhellis3]

I would also add that if we can not dare to do better than our predecessors then why are we even here?

AFAIK, Google only ever used self-play in the Zero training, which means they did not test the method I mentioned above (among many other very clever ideas I am sure many other contributors will come up with). I also answered why they would be inclined to use less efficient methods in my previous posts.

[jjoshua2]

There is a bug now with the time management that got merged in; it didn't make it into the msvc2017 solution, so build fails due to missing those two files, and then once you drag them in, it builds.
Edit: posted this also in the appropriate pull request, which is a better spot for it.

[RavnaBergsndot]

Leela Zero uses a window of 250k games. It then applies symmetry to effectively make the number of trainable games within the window 2 million. The 100k window LC is using is tiny compared to that.

[jjoshua2]

That is a big difference. Did alphazero use 500k with no symmetries?

[jkiliani]

I cannot find any reference to the size of the training window in the AlphaZero paper, they just say they don't use symmetries. This could mean that the window is the same 500k games as for Alphago Zero, or it could mean they use a completely different value. The lack of symmetries is almost certainly detrimental to the result though, since more training data means better generalisation. Google had the resources to simply accept the necessity of more training games as a consequence, but we have to be more economical than that.

I think experimenting with the window size would help, but implementing the symmetries would definitely help in multiple regards: Not only for the training data augmentation, but also since it would allow forgoing Dirichlet noise for evaluation purposes, since games without noise or temperature would then no longer be deterministic (as long as they have different seeds)

[Error323]

Here's my thoughts on this discussion, which we should have now! Good.

Leela Zero uses a window of 250k games. It then applies symmetry to effectively make the number of trainable games within the window 2 million. The 100k window LC is using is tiny compared to that.

Yes, the 100K window was based on a couple of findings/theories:

1. our initial seed data was created from ~150K games with a solid random distribution. My hypothesis is that any net onward would be sampled from this underlying distribution and therefore a smaller window size would be feasible.

2. I tried bigger windows at the start, but the 100K window seemed to work best. I didn't test anything smaller beyond that. Note that 100K games corresponds to ~10M datapoints without applying symmetries and given an average of 100 ply per game.

I think experimenting with the window size would help, but implementing the symmetries would definitely help in multiple regards: Not only for the training data augmentation, but also since it would allow forgoing Dirichlet noise for evaluation purposes, since games without noise or temperature would then no longer be deterministic (as long as they have different seeds)

I very much agree that symmetries would be a good addition! It would allow us to iterate faster. I'd like to work on that once we're fully automated.

@jjoshua2 you switched your own client to do something different, and it's uploading games? I don't think it's a good idea for everyone to have their own versions of the client. We should all only run the officially released client.

I agree here, this is unwise. Even though with the best intentions, we are bound by compute and cannot rerun experiments like this easily. We should be open about everything we do and come to general consensus in discussions here and from that only use the master branch.

I agree we want to be as efficient as possible and get strong as possible ASAP. But I don't see anything in the Google papers that suggests doing matches is more efficient than always promoting. I see the opposite. They published the AGZ paper using matches. Then later they published a second paper where they always promote. I interpret this to mean always promoting is the better method. If it was less efficient why would they switch to it?

I'm also very tempted to use the AlphaZero approach. I think we should give it a go and see if we get good results by monitoring the matches. It'll be sort of a hybrid in that regard. Here are the facts from the AZ paper and some back of the envelope calculations:

1. 44'000'000 games (44M)
2. 700'000 training steps with a batch size of 4096
3. 4 learning rates: [0.2, 0.02, 0.002, 0.0002]
4. 800 playouts

So 44M at our current gamerate per day (30K) means it would take us about 1460 days to repeat this experiment. Given the two symmetry types we could apply it may be reasonable to assume we need 11M games. This means 366 days. I also expect we get more people to help us, let's say we could go up to 60K games per day which means this experiment would run for about 6 months. And I also think we will obtain more efficiency codewise, but this may be countered by (perhaps) requiring bigger neural networks.

My current approach is a little bit different. I've trained the first net using a learning rate of 0.02 and went down to 0.002. Every new net onward always started at 0.02 to get out of potentially local optima. I'd like to suggest we keep doing this. This makes training a bit slower, but we are bound by simulations, not by training. And we can still easily optimize training further (e.g. use multiple GPUs).

But I'm completely open to different approaches. It's our data and compute time in the true sense of the word and we should all have a say in it. What do you think?

[kiudee]

I agree with @Error323 in that we should try the AlphaZero approach with matches for monitoring the progress. If something goes horribly wrong, we can always change it later.

Regarding the learning rates: Could you track the distance the training moves in weight space from initialization (last network weights) to the final weights? This metric would allow us to gauge how the training changes over time.

[Uriopass]

I'm not sure distance in weight space is a very good metric, since even if filters are just swapped the resulting network would be equivalent but very far in weight space.

Computing different kinds of statistics like we have on the stats page, and the opening etc would show better how the behavior changes (for example it hasn't learned to castle properly yet).

[kiudee]

Yes, to evaluate chess knowledge, other statistics are more useful.

The distance in weight space is useful to diagnose effects of the learning rate schedule on the convergence. A small distance might give an indication that the network is stuck in a local optimum and might need a higher initial learning rate.
Of course there are many symmetries in the neural network architecture which will cause the existence of many equivalent local minima, but the metric is more about the question: Did we converge to a similar local minimum as last time, or did we find another one in weight space (even if completely equivalent)?

[jkiliani]

With gen9 and gen10, draws are trending up again after reaching really low levels (7%) in gen8. Could this be related to the slowdown in improvement? Are current nets for instance learning to defend themselves when down in material better than gen8 could?

[evalon32]

@jkiliani Based on what I'm seeing in #92, the current net is notably worse at finding checkmates.

[Uriopass]

~@evalon32 Or... It is better at defending against said checkmates.~
If it was better at defending, it would be better at attacking since it's just selfplay.

[evalon32]

@Uriopass I'll run some tournaments to test that hypothesis when I have time. FWIW, I suspected the same about the KQvK backslide from gen1 to gen2. I was wrong :)

[killerducky]

To clarify what I wrote earlier in this thread about what the Deepmind papers say about matches vs always promote, I think the papers imply always promote is fastest, but I also think the difference is small and either method will work. My objections earlier in the thread were only aimed at how to interpret the papers.

But I think there are advantages to doing matches in our case due to differences in our project vs Deepmind. For example Deepmind had already debugged their process and tuned many parameters. We have a new process that probably still has bugs. Doing matches could expose those bugs. We also have a distributed system and it isn't as easy to tune the rate of games played vs new nets trained.

[jkiliani]

I think 55% gating should also be considered. If the improvement is not statistically significant, it's at least dubious whether adopting the candidate net is a good idea.

[Dorus]

I have no idea why gating gives any advantage in spotting bugs.

Doing match games to draw an elo graph certainly gives an advantage in tuning parameters and spotting potential problems when progress unexpectedly halts. However, gating doesn't add much to this, gating is purely a way to select new networks that might harm overall progress as much as it could help.

I have no opinion on if it will harm of benefit the project, just that AZ didn't use gating and AGZ did. Btw i think they did draw an elo graph for AZ so there is that.

[jhellis3]

One benefit of 50% is more promotions in general, which should mean a greater variety of play over a large set of games. I would think this would, overall, be beneficial. Eventually, we may want to ensure some minimal quality of the latest net for those seeking to use LCZ as they would any other chess engine. If so, then the games number per match will need to be raised along with some minimal score % or SPRT result. If we only play the strongest net against itself and only promote on measured improvement, game variety will decline significantly over time, which could slow continued progress.

[jkiliani]

I guess there are good arguments for promoting at 50%, but I'm not worried about game variety honestly. Using t=1 for the whole game tends to ensure that already very effectively.

[jhellis3]

Sure it is effective at lower Elos, but I am not sure how good relying on t=1 at very high Elos will be. Yes, you will get more variety, but engines at that level tend to punish inaccuracies of any significant amount quite quickly and severely. So while it might continue to provide variety, the % of genuinely useful data contained in that variety may decline over time.

[davidspencer6174]

The good thing about always-promote is that strategies that take more than one generation to understand can still arise. I talked about dragons in the corresponding LZ thread, and I think a good example in chess is the IQP.

IQP positions are often sharp and challenging to play, particularly for the player with the pawn, who needs to play forcefully. If this is not LCZ's natural style, it may sometimes try out IQP positions, but lose them because it doesn't know how to follow up. We would like it to learn how to play IQP positions better. However, in a gating system, we may end up choosing networks that get into fewer IQP positions rather than networks that are actually better at them, since the IQP side will lose more often at first. This artificial selection of networks that get into fewer IQP positions wouldn't occur in always-promote, so it would have a better chance of learning how to play them well.

I can certainly see that there are some benefits to gating as well - the excitement of match games bringing people to the project is a big one. Still, I wanted to point out this kind of situation, which I think could hamper progress once LCZ becomes extremely strong.

[kiudee]

I agree with @davidspencer6174 that it could take the network several generations until a learned pattern can reliably be turned into more wins.
By gating we are implicitly regularizing the network to functions which immediately manifest into a gain in strength. Not sure which effect this has in the long run.

edit: The AZ always-promote method ensures that the network replicates the MCTS policy as closely as possible. Thus even if the networks is inferior, it should provide useful self-play data.

[jjoshua2]

I think a minimal level of gating is good, to keep disastrous networks from making too much noise, and also at the end doing a selection of lots of similarly trained at different steps networks will provide a small elo boost simply by survival of the fittest. But switching to a bigger blocksize might be a better use of resources instead of getting the last few elo by using testing and hybrids.

[amj]

Hi! I've tried to reproduce AGZ in the Minigo project. As i was very resource constrained, i also opted for "always promote" and it hasn't worked out the way i thought. I wrote a bit about it here

The short version is that "always promote" is most useful when you already have other indicators you can monitor to be sure you're "going somewhere." If you don't have those, then gating promotion is a good check on the overall health. For LZ, that was already demonstrated, and they were able to use 'time between promotion' as an indicator for other things, like when to cut the learning rate or when to expand the size of the network.

The most useful (non-promotion) indicators to monitor are 1. Selfplay rating improvement (ideally not Elo, but a Bradley-Terry model), 2. Performance on a non-selfplay dataset, 3. Error on a holdout set of games not used in training.

For us, (2) meant a validation set of pro games where we can see the value-error and policy error by model.

LCZero has an easy way to build a good dataset for (2), as there's plenty of pro chess games.

It seems like DM was able to take away promotion matches because they were already confident in the rest of their monitoring indicators. If you don't have those indicators, it's probably not going to 'just work' the first time.

[jkiliani]

I think the point @davidspencer6174 raised only becomes really relevant in case a whole training window is passed with no promotion at all. Otherwise, the newly learned knowledge from the self-play games since the previous promotion will be learned all in one lot once the next promotion occurs, rather than in several batches. This may slow down the feedback somewhat, but protect against overfit networks which I think should be worth the cost.

When we do come to the point where a whole training window passes with no promotion, maybe further increasing the window would be an option. At some point we'll have to do a bootstrap to a bigger network but I think this will be quite a while yet.

[davidspencer6174]

@kiudee Your second sentence is a great, precise way of articulating the idea I was trying to get at. Thank you for posting that.

@amj Thanks, definitely a lot of food for thought there. I should clarify that I am advocating for always-promote more out of a belief that it could lead to a stronger network, rather than out of a desire to save resources. But I had not considered the issue of figuring out how to tune the hyperparameters without having test matches. Personally, I think the idea you mention in your linked post of "do[ing] the evaluation games as the holdout data, without using it as a gating function for promotion" is a great solution.

@jkiliani I agree that over time it may be necessary to increase the training window, but to me that seems like a separate issue. I don't see any way that changing the training window size could combat the potential issues that might arise from implicit regularization.

[jjoshua2]

https://lichess.org/p7W6goyS gen 12 does what gen 11 could not against SF 9 level 20 3 ply
4 ply https://lichess.org/ZzHXhBcn#40

[glinscott]

@jjoshua2 wow, that's a fun game, lczero trapped the queen in a beautiful windmill. Full strength SF thinks the best move is to actually trade the queen for the knight and bishop, but even then white has a big advantage.

[Dorus]

I just noticed 9c9970d4 failed to produce any self play games. Is this because it failed the match? If so that means gating is now applied, but also t=1 is used all game. This is a bit of a mix of AGZ that used gating and t=1 only 30 moves, and AZ that used t=1 all game and always promote.

Another part of gating is that you go back to the current best if no promotion happens for a bunch of networks. Will this be applied too?

Not that gating with t=1 all game is bad, it could be a good new experiment (but for an experiment you would eventually also have to go back to the situation where 9c9970d4 was created, and run always promote from that point, to see if it gains elo faster or slower overall). Or we just make a gut call and use this combination.

Myself i'm a fan of going full AZ and use always promote, and then still run match games vs current best to draw an elo graph.

[jjoshua2]

https://lichess.org/6uZduZbm 2 min/move vs 5 ply with a client or two running at the same time. I edited previous post to put the 4 ply result in... Both gen12. Running id18 on 6 ply now

[Error323]

I've finished the automated training and uploading code. It just uploaded the first net. Should have a new net every 4hrs or so.

[CMCanavessi]

Just a small detail Error323, the network name in the new .gz is not truncated as before. Maybe that was intentional though?

[Error323]

Yes, that was intentional. I didn't want to risk a clash during upload.

What was not intentional is that the trainingcode has sucked up 98% of my RAM. System hangs...

[amj]

@Error323 memory leak?