Leela-chess: Different elo targets
The bottom elo on ccrl 40/4 is only 276 elo and sometimes loses to a random mover, but it requires java.
A good first target might be chessputer open source UCI cpp at 765 elo.
I don't know an elo, but alan turing's historic chess program has been implemented in chessbase engine UCI (download), and played against Kasprov (he beat in 16 moves). Would be good publicity, and it can be set to different ply depths.
Robocide, open source C UCI engine, 1897 elo
Ruffian 2.1.0 rated 2609. Was the best free engine I used to use a long time ago.
Crafty, famous, elo 2400-3000 depending on version.
Scorpio 2.7.9 was the weakest engine in the bottom TCEC 4th league around 2900 elo.
Gull 3, a strong open source program, now mid-level TCEC 1st league around 3200 elo.
Andsacs .93 open source mid-level TCEC Premier league 3300 elo with 4 CPU.
Komodo 9, winner of TCEC 8, now free, 3383 4 CPU.
Stockfish 9 top released engine, open source, 3560 elo with 4 CPU.
jjoshua2
All 133 comments
I tried to load lczero as UCI engine to match some of these and couldn't get it to work, even when I hardcoded the -w weights file, since GUIs don't let you send params. Maybe it takes too long to initialize or would work in other gui?
jjoshua2
on 13 Mar 2018
Which GUI are you using? The cutechess gui should let you pass command line parameters.
glinscott
on 13 Mar 2018
I was using infinity chess gui. I can try cutechess I suppose. Chessbase
also don't support command line. I think the arguments really should be uci
parameters. Especially threads.
On Mar 12, 2018 11:47 PM, "Gary Linscott" notifications@github.com wrote:
Which GUI are you using? The cutechess gui should let you pass command
line parameters.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/glinscott/leela-chess/issues/109#issuecomment-372537508,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AO6INOvEzFdSqVs62Kb8foTJUFesckZjks5td0E0gaJpZM4SnVT3
.
jjoshua2
on 13 Mar 2018
009 - Qualy League Ratings.txt
009 - Qualy League Ratings Head2head.txt
I will test Gen 5 in a couple of days.
CMCanavessi
on 13 Mar 2018
Once it starts getting stronger, agreed, UCI parameters make sense. Or if someone else sends a PR I'd be happy to merge it :).
glinscott
on 13 Mar 2018
I would be happy with at least some Time Control management. I wanted to make some 30''+0.5'' tournaments but it was constantly losing on time, as it takes around 1.7 seconds to move in my box, using default 800 playouts.
CMCanavessi
on 13 Mar 2018
I did a number of round-robin tournaments with Stockfish at different Skill levels and constant time control 1 sec / 40 moves, as well as with (slightly) longer time controls, to get an idea on its scaling. Here are my results:
Rank Name Elo +/- Games Score Draws
1 sf 861 136 1000 99.3% 0.6%
2 sf10 175 24 1000 73.2% 2.6%
3 sf8 114 22 1000 65.8% 2.4%
4 sf5 -90 22 1000 37.4% 1.9%
5 sf3 -258 27 1000 18.4% 1.5%
6 sf1 -481 46 1000 5.9% 0.4%
sf meaning Stockfish without Skill level setting, all engines 40/1
Rank Name Elo +/- Games Score Draws
1 sf20 531 114 200 95.5% 4.0%
2 sf 449 93 200 93.0% 4.0%
3 sf17 173 51 200 73.0% 10.0%
4 sf16 160 52 200 71.5% 5.0%
5 sf19 149 51 200 70.3% 7.5%
6 sf18 149 51 200 70.3% 7.5%
7 sf15 106 49 200 64.8% 7.5%
8 sf13 89 47 200 62.5% 10.0%
9 sf14 70 47 200 60.0% 9.0%
10 sf12 63 45 200 59.0% 14.0%
11 sf11 47 46 200 56.8% 10.5%
12 sf9 42 46 200 56.0% 11.0%
13 sf10 37 47 200 55.3% 8.5%
14 sf8 -12 46 200 48.3% 8.5%
15 sf7 -133 51 200 31.8% 3.5%
16 sf6 -166 53 200 27.8% 4.5%
17 sf5 -246 62 200 19.5% 1.0%
18 sf4 -279 65 200 16.8% 1.5%
19 sf3 -382 83 200 10.0% 1.0%
20 sf2 -470 109 200 6.3% 0.5%
21 sf1 -676 363 200 2.0% 0.0%
Dito, it seems Skill level=20 is equivalent to not setting a skill level?
Rank Name Elo +/- Games Score Draws
1 sf9 492 70 450 94.4% 1.8%
2 sf8 287 43 450 83.9% 3.8%
3 sf7 226 39 450 78.6% 2.4%
4 sf6 112 33 450 65.6% 4.0%
5 sf5 39 32 450 55.6% 3.1%
6 sf4 -31 32 450 45.6% 1.8%
7 sf3 -140 34 450 30.9% 2.7%
8 sf2 -241 40 450 20.0% 2.2%
9 sf1 -303 45 450 14.9% 1.3%
10 sf0 -369 52 450 10.7% 0.9%
I wasn't aware that 0 was a valid setting for Skill level until then, but tried it at this point. All games 1 sec / 40 moves.
Rank Name Elo +/- Games Score Draws
1 sf_40/16 350 53 200 88.3% 21.5%
2 sf_40/8 184 43 200 74.3% 30.5%
3 sf_40/4 12 42 200 51.7% 25.5%
4 sf_40/2 -151 44 200 29.5% 25.0%
5 sf_40/1 -470 84 200 6.3% 8.5%
And finally, a time scaling test that revealed very considerable scaling at such short time controls.
I just measured @Error323's supervised net kbb1-64x6-796000.txt against sf5, i.e.
./cutechess-cli -rounds 70 -tournament gauntlet -concurrency 2 -pgnout SF0.pgn \
-engine name=lc_kbb1 cmd=lczero arg="--threads=1" arg="--weights=$WDR/kbb1-64x6-796000.txt" arg="--playouts=800" arg="--noponder" arg="--noise" tc=inf \
-engine name=sf_lv5 cmd=stockfish_x86-64 option.Threads=1 option."Skill Level"=5 tc=40/1 \
-each proto=uci
The result:
Score of lc_kbb1 vs sf_lv5: 39 - 29 - 2 [0.571] 70
Elo difference: 49.98 +/- 82.52
From the previous tests, sf5 should be roughly 450 Elo above sf0. So far, I had none of the reinforcement learning nets scoring any wins or draws against sf0 yet, but I'm going to run another match tonight with gen6 to test whether that changes today. I'll update when I have something.
jkiliani
on 13 Mar 2018
@jkiliani very cool, thanks for the numbers. At some point, we can do a round-robin tournament and get a better idea of the overall ELO progression as well.
glinscott
on 13 Mar 2018
I think @CMCanavessi tournament is more interesting using engines of similar strength. Crippling a strong engine doesn't make as much sense to me, but I am always excited to see both, so keep them coming! SF skill level increasing is easier and more efficient to test than against many engines though.
I can't wait to see how gen6 does. I think it will be about 600 elo. I estimated gen4 was 530 between Acqua and NEG on CCRL 40/4. Although seeing 412 with a 0 elo random mover was good too. CCRL base is Brutus RND at 200 elo.
jjoshua2
on 13 Mar 2018
Running a tournament right now with 24 engines under ~1000 elo, Leela Chess Zero Gen 6 is playing. Will update later with how it's doing. Should do somewhat better than Gen 4, but can it beat NEG and Acqua consistently now? Can it at least draw with Easy Peasy? We'll see...
CMCanavessi
on 13 Mar 2018
@CMCanavessi you can reuse most of the engine v engine games from prior tournaments and just rerun LCZ right? is that the strategy you are doing?
jjoshua2
on 13 Mar 2018
Yes, you can do that but I'm just running a completely new tournament, with more rivals. Looking good so far.
In the previous tournament, LCZ Gen 4 played 10 games vs. NEG and got 0 wins, 1 draw and 9 loses. In this tournament it has already played once vs NEG and it won. And seeing it play, it looks to have much better endgame undesrtanding. We'll see how it looks when more games are played, so far it's at 50%.
CMCanavessi
on 13 Mar 2018
Did some research and it looks like NEG and Acqua both just do a 1 ply search with no lookahead, so it should be easy for a neural network + any search to beat them once the net understands the very basics. It may be interesting if you could put an SF with skill=0 in your tournament. I think it might be a 1 ply plus quiescent search? I think I saw skill=1 was 3 ply.
Edit: it appears skill level has maxDepth of level + 1, but also randomly picks among the top 4 moves as long as its not a major blunder.
jjoshua2
on 13 Mar 2018
1st round robin out of 10 has been played and Gen 6 is showing better performance compared to Gen 4. Here are the current standings:
Engine Score BRXaUsIoTaHaMFNSYoZoPyDiPyAcN.LeEaRaLaCPPOEtEtTe S-B
01: BRAMA 05/12/2004 x32 20.5/23 · 0 1 1 0 = 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 209,25
02: Xadreco 5.83 x32 19.5/23 1 · 0 = 1 = 1 = = 1 1 = 1 1 1 1 1 1 1 1 1 1 1 1 198,50
03: Usurpator II x32 19.0/23 0 1 · 1 1 = 0 1 1 0 1 = 1 1 1 1 1 1 1 1 1 1 1 1 190,75
04: Iota 1.0 x32 18.5/23 0 = 0 · = 1 = 1 1 1 1 1 1 1 1 1 1 = 1 1 1 = 1 1 185,50
05: Talvmenni 0.1 x32 18.0/23 1 0 0 = · = = = 1 = 1 1 1 1 = 1 1 1 1 1 1 1 1 1 173,75
06: Hanzo the Razor x32 17.5/23 = = = 0 = · = 1 1 = = = 1 1 = 1 1 1 1 1 1 1 1 1 168,50
07: MFChess 1.3 x32 17.0/23 0 0 1 = = = · = = = 1 1 1 1 1 1 = = 1 1 1 1 1 1 162,00
08: NSVChess 0.14 x32 15.0/23 0 = 0 0 = 0 = · = 1 = = = 1 1 1 1 1 1 1 = 1 1 1 130,25
09: Youk V1.05 x32 15.0/23 0 = 0 0 0 0 = = · = = = 1 1 1 1 1 1 1 1 1 1 1 1 122,00
10: Zoe 0.1 x32 14.0/23 0 0 1 0 = = = 0 = · 0 = = 0 1 1 1 1 1 1 1 1 1 1 119,50
11: Pyotr Amateur Edition v0.6 x32 13.5/23 0 0 0 0 0 = 0 = = 1 · = = 1 0 1 1 1 1 1 1 1 1 1 105,00
12: Dikabi v0.4209 x32 12.0/23 0 = = 0 0 = 0 = = = = · = = 1 = 1 = = 1 = 1 1 = 109,00
13: Pyotr Novice Edition v2.6 x32 11.5/23 0 0 0 0 0 0 0 = 0 = = = · 1 1 0 1 1 = 1 1 1 1 1 80,75
14: Acqua ver. 20160918 x32 10.0/23 0 0 0 0 0 0 0 0 0 1 0 = 0 · 0 1 1 1 = 1 1 1 1 1 63,00
15: N.E.G. 1.2 x32 9.5/23 0 0 0 0 = = 0 0 0 0 1 0 0 1 · 0 = = 1 1 = 1 1 1 67,75
16: Leela Chess Zero Gen 6 x64 9.0/23 0 0 0 0 0 0 0 0 0 0 0 = 1 0 1 · 0 1 = 1 1 1 1 1 52,50
17: Easy Peasy 1.0 x32 8.5/23 0 0 0 0 0 0 = 0 0 0 0 0 0 0 = 1 · = 1 1 1 1 1 1 46,50
18: Ram 2.0 x32 7.5/23 0 0 0 = 0 0 = 0 0 0 0 = 0 0 = 0 = · = = 1 1 1 1 48,25
19: LaMoSca v0.10 x32 5.0/23 0 0 0 0 0 0 0 0 0 0 0 = = = 0 = 0 = · = = = = = 32,75
20: CPP1 0.1038 x32 5.0/23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = = · 1 1 1 1 16,75
21: POS v1.20 x32 4.0/23 0 0 0 0 0 0 0 = 0 0 0 = 0 0 = 0 0 0 = 0 · = 1 = 25,00
22: EtherTrueRand 9.21 x64 3.0/23 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 = · = 1 16,25
23: EtherealRandom (8.97) x64 2.0/23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 = · 1 5,50
24: Teki Random Mover x64 1.5/23 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 = 0 = 0 0 · 10,50
276 of 2760 games played
And here's how the bottom of my rating list looks like right now:
156 MFChess 1.3 x32 : 1001.0 23 13 8 2 74 35 661.7 23 23.0
157 Hippocampe v0.4.2 x32 : 982.0 150 98 18 34 71 12 652.1 15 15.0
158 Youk V1.05 x32 : 958.4 62 16 7 39 31 11 1204.8 45 42.0
159 NSVChess 0.14 x32 : 860.7 173 92 39 42 64 23 661.2 29 19.7
160 Zoe 0.1 x32 : 833.9 23 11 6 6 61 26 669.0 23 23.0
161 Pyotr Amateur Edition v0.6 x32 : 814.4 23 11 5 7 59 22 669.8 23 23.0
162 Easy Peasy 1.0 x32 : 756.6 173 92 16 65 58 9 667.8 29 19.7
163 Dikabi v0.4209 x32 : 737.3 23 5 14 4 52 61 673.2 23 23.0
164 Pyotr Novice Edition v2.6 x32 : 708.6 23 9 5 9 50 22 674.4 23 23.0
165 N.E.G. 1.2 x32 : 574.4 173 70 17 86 45 10 679.4 29 19.7
166 Acqua ver. 20160918 x32 : 569.7 173 72 12 89 45 7 679.7 29 19.7
167 Leela Chess Zero Gen 6 x64 : 568.9 23 8 2 13 39 9 680.5 23 23.0
168 Leela Chess Zero Gen 4 x64 : 413.8 150 43 18 89 35 12 690.0 15 15.0
169 Ram 2.0 x32 : 413.6 173 44 29 100 34 17 689.6 29 19.7
170 CPP1 0.1038 x32 : 361.6 173 35 34 104 30 20 692.9 29 19.7
171 LaMoSca v0.10 x32 : 282.3 173 1 83 89 25 48 698.0 29 19.7
172 POS v1.20 x32 : 165.7 173 13 33 127 17 19 705.4 29 19.7
173 EtherTrueRand 9.21 x64 : 50.6 173 2 33 138 11 19 712.7 29 19.7
174 EtherealRandom (8.97) x64 : 35.5 23 1 2 20 9 9 703.7 23 23.0
175 Teki Random Mover x64 : 0.0 173 0 29 144 8 17 715.9 29 19.7
Too early to quantify the gain, but Gen6 is clearly stronger than Gen4. We'll see tomorrow when a couple more rounds are played.
CMCanavessi
on 14 Mar 2018
Leela Gen 6 has played 55 games now in the new tournament, and things look much better than Gen 4. here's the updated ratings from the bottom of my rating list:
152 Usurpator II x32 : 1019.6 55 40 5 10 77 9 653.7 23 22.5
153 Talvmenni 0.1 x32 : 998.7 55 34 16 5 76 29 649.2 23 22.5
154 StrategicDeep 1.25 x32 : 989.6 39 3 2 34 10 5 1501.8 23 22.1
155 Hanzo the Razor x32 : 981.9 55 30 24 1 76 44 626.8 23 22.5
156 MFChess 1.3 x32 : 954.1 55 31 17 7 72 31 653.0 23 22.5
157 Hippocampe v0.4.2 x32 : 933.4 150 98 18 34 71 12 618.0 15 15.0
158 Youk V1.05 x32 : 918.2 94 38 10 46 46 11 975.0 45 42.8
159 Zoe 0.1 x32 : 818.4 55 28 14 13 64 25 628.8 23 22.5
160 NSVChess 0.14 x32 : 800.5 205 103 54 48 63 26 626.3 29 22.7
161 Pyotr Amateur Edition v0.6 x32 : 787.7 55 26 16 13 62 29 616.2 23 22.5
162 Dikabi v0.4209 x32 : 740.6 55 14 34 7 56 62 633.3 23 22.5
163 Easy Peasy 1.0 x32 : 683.5 205 102 22 81 55 11 636.9 29 23.1
164 Pyotr Novice Edition v2.6 x32 : 613.6 55 19 11 25 45 20 654.0 23 22.5
165 Leela Chess Zero Gen 6 x64 : 587.8 55 18 12 25 44 22 638.3 23 22.5
166 N.E.G. 1.2 x32 : 532.5 205 77 24 104 43 12 652.7 29 23.6
167 Acqua ver. 20160918 x32 : 527.7 205 82 15 108 44 7 646.2 29 23.1
168 Ram 2.0 x32 : 391.8 205 50 38 117 34 19 650.8 29 22.5
169 Leela Chess Zero Gen 4 x64 : 383.9 150 43 18 89 35 12 654.6 15 15.0
170 CPP1 0.1038 x32 : 331.7 205 39 43 123 30 21 651.9 29 22.9
171 LaMoSca v0.10 x32 : 271.2 205 2 99 104 25 48 658.1 29 22.7
172 POS v1.20 x32 : 153.2 205 15 39 151 17 19 674.6 29 23.4
173 EtherealRandom (8.97) x64 : 65.7 55 2 8 45 11 15 656.3 23 22.5
174 EtherTrueRand 9.21 x64 : 40.1 205 2 40 163 11 20 677.8 29 23.2
175 Teki Random Mover x64 : 0.0 205 0 36 169 9 18 675.2 29 22.7
Now we can start to see those +200 elo stronger showing. After this tournament is finished (still many games to go) I'll probably wait for Gen 10 or something like that to make a new test.
CMCanavessi
on 14 Mar 2018
Thanks! I would appreciate even if you could just run a gauntlet of Gen 7 against Easy Peasy Pyotr and Neg. There is debate about how inflated the +200 elo in self play is with it not seeming to gain much yet against SF level 0.
jjoshua2
on 15 Mar 2018
Gen 8 :)
Error323
on 15 Mar 2018
Here are the final standings of the tournament with Gen 6
Engine Score BR Io Xa Ta Us Ha MF Yo Zo Py NS Di Py Ea Le N. Ac Ra CP La PO Et Et Te S-B
01: BRAMA 05/12/2004 x32 81.5/92 ···· 1==1 01=1 0==0 111= ==1= 1==1 1111 1111 11=1 111= 1=1= 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 3370,2
02: Iota 1.0 x32 76.0/92 0==0 ···· =101 ===0 0=01 1=== =1=1 1111 1=1= 1=11 1111 1111 1111 1111 11=1 1111 1111 =1=1 1111 1111 1111 =11= 1111 1111 3075,0
03: Xadreco 5.83 x32 75.5/92 10=0 =010 ···· 11=0 0101 ==0= 1111 =1=0 1111 1011 =11= ==1= 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 2981,5
04: Talvmenni 0.1 x32 73.5/92 1==1 ===1 00=1 ···· 001= ==== ==0= 111= ==1= 1111 ===1 1=1= 1111 111= 1111 =111 1111 11=1 1111 1111 11=1 1111 1111 1111 2946,7
05: Usurpator II x32 71.5/92 000= 1=10 1010 110= ···· =0=0 0001 1110 0101 1110 1=11 ===1 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 2725,5
06: Hanzo the Razor x32 70.0/92 ==0= 0=== ==1= ==== =1=1 ···· ===1 111= ==== =1== 1=1= ==== 1111 11== 1==1 =111 1111 11=1 1111 1111 1111 11=1 1111 1=11 2807,0
07: MFChess 1.3 x32 66.0/92 0==0 =0=0 0000 ==1= 1110 ===0 ···· =0== =1== 1=11 ==== 10== 11=1 =111 1=11 1111 1111 =1=1 1111 1111 1111 1111 1111 1111 2448,5
08: Youk V1.05 x32 64.5/92 0000 0000 =0=1 000= 0001 000= =1== ···· =11= ===1 =11= ==1= 111= 11=1 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 2251,7
09: Zoe 0.1 x32 58.0/92 0000 0=0= 0000 ==0= 1010 ==== =0== =00= ···· 0110 0=== =01= =1== 11== 1=11 1111 0111 11=1 1111 1111 1111 1111 1111 1111 1986,2
10: Pyotr Amateur Edition v0.6 x32 54.5/92 00=0 0=00 0100 0000 0001 =0== 0=00 ===0 1001 ···· ==== ==== =1=0 11=1 1=11 01=1 1111 1=11 1111 111= 1111 1111 1111 1=11 1813,0
11: NSVChess 0.14 x32 52.0/92 000= 0000 =00= ===0 0=00 0=0= ==== =00= 1=== ==== ···· ==0= ===0 1=10 1==0 1111 1==1 1111 1=11 1==1 =11= 1111 1111 1111 1767,7
12: Dikabi v0.4209 x32 50.0/92 0=0= 0000 ==0= 0=0= ===0 ==== 01== ==0= =10= ==== ==1= ···· =111 1=11 ==== 11== =1=1 ==== 1=== =111 ==1= 1=== 1=== ===1 1981,7
13: Pyotr Novice Edition v2.6 x32 45.0/92 0000 0000 0000 0000 0000 0000 00=0 000= =0== =0=1 ===1 =000 ···· 11=1 0==1 1==1 1110 1=11 111= ==11 1111 1111 1111 1111 1264,7
14: Easy Peasy 1.0 x32 40.5/92 0000 0000 0000 000= 0000 00== =000 00=0 00== 00=0 0=01 0=00 00=0 ···· 1101 =1=1 0011 =111 1111 1=== 1111 1111 1111 1111 1096,2
15: Leela Chess Zero Gen 6 x64 40.0/92 0000 00=0 0000 0000 0000 0==0 0=00 0000 0=00 0=00 0==1 ==== 1==0 0010 ···· 10== 0011 1111 1110 ==11 1111 1111 1111 1111 1093,2
16: N.E.G. 1.2 x32 34.5/92 0000 0000 0000 =000 0000 =000 0000 0000 0000 10=0 0000 00== 0==0 =0=0 01== ···· 1101 ==01 1=11 1=1= =111 1111 1111 1111 839,75
17: Acqua ver. 20160918 x32 34.5/92 0000 0000 0000 0000 0000 0000 0000 0000 1000 0000 0==0 =0=0 0001 1100 1100 0010 ···· 11=1 1101 ==11 1111 1111 1111 1111 809,00
18: Ram 2.0 x32 30.0/92 0000 =0=0 0000 00=0 0000 00=0 =0=0 0000 00=0 0=00 0000 ==== 0=00 =000 0000 ==10 00=0 ···· ==1= ==== 1111 1111 1111 1111 771,25
19: CPP1 0.1038 x32 22.5/92 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0=00 0=== 000= 0000 0001 0=00 0010 ==0= ···· ==== 1=11 111= 1=11 111= 454,25
20: LaMoSca v0.10 x32 20.0/92 0000 0000 0000 0000 0000 0000 0000 0000 0000 000= 0==0 =000 ==00 0=== ==00 0=0= ==00 ==== ==== ···· ==== ==== ==== ==1= 516,50
21: POS v1.20 x32 15.0/92 0000 0000 0000 00=0 0000 0000 0000 0000 0000 0000 =00= ==0= 0000 0000 0000 =000 0000 0000 0=00 ==== ···· =1=1 11=1 ===1 319,50
22: EtherTrueRand 9.21 x64 10.5/92 0000 =00= 0000 0000 0000 00=0 0000 0000 0000 0000 0000 0=== 0000 0000 0000 0000 0000 0000 000= ==== =0=0 ···· =0== 1=== 289,00
23: EtherealRandom (8.97) x64 9.5/92 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0=== 0000 0000 0000 0000 0000 0000 0=00 ==== 00=0 =1== ···· 1=== 182,50
24: Teki Random Mover x64 9.0/92 0000 0000 0000 0000 0000 0=00 0000 0000 0000 0=00 0000 ===0 0000 0000 0000 0000 0000 0000 000= ==0= ===0 0=== 0=== ···· 231,00
1104 games played / Tournament finished
And here's how the rating list currently stands:
152 Safrad 2.1.35.210 x32 : 1007.4 208 112 23 73 59 11 825.7 35 27.9
153 Usurpator II x32 : 991.9 92 67 9 16 78 10 620.1 23 23.0
154 Hanzo the Razor x32 : 970.9 92 50 40 2 76 43 621.1 23 23.0
155 StrategicDeep 1.25 x32 : 918.9 58 4 3 51 9 5 1443.0 23 22.5
156 MFChess 1.3 x32 : 917.1 92 52 28 12 72 30 623.4 23 23.0
157 Youk V1.05 x32 : 916.8 150 61 21 68 48 14 938.2 45 43.4
158 Hippocampe v0.4.2 x32 : 901.1 150 98 18 34 71 12 596.8 15 15.0
159 Zoe 0.1 x32 : 814.9 92 45 26 21 63 28 627.8 23 23.0
160 Pyotr Amateur Edition v0.6 x32 : 771.1 92 42 25 25 59 27 629.7 23 23.0
161 NSVChess 0.14 x32 : 761.3 242 114 69 59 61 29 615.3 29 25.1
162 Dikabi v0.4209 x32 : 714.6 92 21 58 13 54 63 632.2 23 23.0
163 Easy Peasy 1.0 x32 : 667.9 242 117 30 95 55 12 620.7 29 25.1
164 Pyotr Novice Edition v2.6 x32 : 650.6 92 35 20 37 49 22 635.0 23 23.0
165 Leela Chess Zero Gen 6 x64 : 584.4 92 31 18 43 43 20 637.9 23 23.0
166 N.E.G. 1.2 x32 : 511.9 242 89 29 124 43 12 629.7 29 25.1
167 Acqua ver. 20160918 x32 : 506.4 242 94 17 131 42 7 630.0 29 25.1
168 Ram 2.0 x32 : 388.8 242 58 46 138 33 19 636.8 29 25.1
169 Leela Chess Zero Gen 4 x64 : 369.4 150 43 18 89 35 12 632.3 15 15.0
170 CPP1 0.1038 x32 : 323.9 242 45 49 148 29 20 640.6 29 25.1
171 LaMoSca v0.10 x32 : 253.3 242 2 111 129 24 46 644.7 29 25.1
172 POS v1.20 x32 : 144.0 242 18 45 179 17 19 651.0 29 25.1
173 EtherealRandom (8.97) x64 : 52.3 92 2 15 75 10 16 661.0 23 23.0
174 EtherTrueRand 9.21 x64 : 34.8 242 2 48 192 11 20 657.3 29 25.1
175 Teki Random Mover x64 : 0.0 242 0 44 198 9 18 659.3 29 25.1
You can see that Gen 6 is about 215 elo stronger than Gen 4.
I will test Gen 8 later today.
CMCanavessi
on 15 Mar 2018
I have started a gauntlet, Leela Gen 8 vs all 23 engines that Gen 6 played against. 4 rounds, 92 games total. We'll see the real improvement in a couple of hours.
CMCanavessi
on 15 Mar 2018
Leela just beat Pyotr Novice Edition in 16 moves... I'm absolutely impressed with Gen 8. It's playing MUCH better than Gen 6. It looks like it knows what it's doing now. I makes logic moves and plays with some kind of sense. It's difficult to explain.
It still has some trouble with endgames, it will shuffle and shuffle for several movements before mating, even with 5 queens vs lonely king haha.
CMCanavessi
on 15 Mar 2018
Awesome! Can you post perhaps one of the interesting games as gif here?
Error323
on 15 Mar 2018
How do I do that? I can post the full pgn if needed.
CMCanavessi
on 15 Mar 2018
@kiudee has a nice tool. I think he uses lichess? I used this one http://www.apronus.com/chess/wbeditor.php
Error323
on 15 Mar 2018
I used the PGN editor on caissa.com for the animations.
kiudee
on 15 Mar 2018
I'd prefer we find a solution that includes pgn files. If someone posts games that need debugging we need pgn to input it into lzchess. http://eidogo.com/ links are the standard for Go, is there not something similar for Chess where you can post links to a game viewer that allows pgn downloads?
killerducky
on 15 Mar 2018
Just for comparison, Gen 6 got 40 points in 92 games in this gauntlet. Gen 8 already has 26.5 points in 46 games.
Estimated elo so far:
# PLAYER : RATING PLAYED W D L (%) D(%) OppAvg OppN OppDiv
163 Usurpator II x32 : 1040.1 105 73 11 21 75 10 690.9 34 30.0
164 Safrad 2.1.35.210 x32 : 998.4 208 112 23 73 59 11 817.4 35 27.9
165 Hanzo the Razor x32 : 986.1 94 51 41 2 76 44 633.4 24 23.8
166 MFChess 1.3 x32 : 932.7 94 53 29 12 72 31 635.6 24 23.8
167 Youk V1.05 x32 : 925.2 152 62 22 68 48 14 933.9 46 44.3
168 StrategicDeep 1.25 x32 : 907.6 58 4 3 51 9 5 1423.2 23 22.5
169 Hippocampe v0.4.2 x32 : 896.5 150 98 18 34 71 12 592.9 15 15.0
170 Zoe 0.1 x32 : 831.6 94 46 27 21 63 29 639.9 24 23.8
171 Pyotr Amateur Edition v0.6 x32 : 782.1 94 42 27 25 59 29 642.0 24 23.8
172 Leela Chess Zero Gen 8 x64 : 775.4 46 23 7 16 58 15 647.4 23 23.0
173 NSVChess 0.14 x32 : 765.2 244 115 69 60 61 28 617.5 30 25.6
174 Dikabi v0.4209 x32 : 732.6 94 22 59 13 55 63 644.2 24 23.8
175 Easy Peasy 1.0 x32 : 667.0 244 117 30 97 54 12 623.1 30 25.6
176 Pyotr Novice Edition v2.6 x32 : 650.4 94 35 20 39 48 21 647.6 24 23.8
177 Leela Chess Zero Gen 6 x64 : 591.6 92 31 18 43 43 20 647.4 23 23.0
178 N.E.G. 1.2 x32 : 511.6 244 89 29 126 42 12 632.0 30 25.6
179 Acqua ver. 20160918 x32 : 506.2 244 94 17 133 42 7 632.3 30 25.6
180 Ram 2.0 x32 : 388.7 244 58 46 140 33 19 639.1 30 25.6
181 Leela Chess Zero Gen 4 x64 : 369.1 150 43 18 89 35 12 628.0 15 15.0
182 CPP1 0.1038 x32 : 323.9 244 45 49 150 28 20 642.8 30 25.6
183 LaMoSca v0.10 x32 : 253.3 244 2 111 131 24 45 646.9 30 25.6
184 POS v1.20 x32 : 144.0 244 18 45 181 17 18 653.1 30 25.6
185 EtherealRandom (8.97) x64 : 52.5 94 2 15 77 10 16 673.1 24 23.8
186 EtherTrueRand 9.21 x64 : 34.8 244 2 48 194 11 20 659.4 30 25.6
187 Teki Random Mover x64 : 0.0 244 0 44 200 9 18 661.4 30 25.6
At the end of round 3, Gen 8 already has 39.5 points, only 0.5 less than Gen 6 got with 1 more full round played. That's the kind of improvement we got :D
Last round starting now, will post results in a while.
CMCanavessi
on 15 Mar 2018
Here's the finished gauntlet:
-----------------Leela Chess Zero Gen 8 x64-----------------
Leela Chess Zero Gen 8 x64 - Acqua ver. 20160918 x32 : 3,0/4 3-1-0 (1101) 75% +191
Leela Chess Zero Gen 8 x64 - BRAMA 05/12/2004 x32 : 0,5/4 0-3-1 (000=) 13% -330
Leela Chess Zero Gen 8 x64 - CPP1 0.1038 x32 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 8 x64 - Dikabi v0.4209 x32 : 1,5/4 1-2-1 (0=10) 38% -85
Leela Chess Zero Gen 8 x64 - Easy Peasy 1.0 x32 : 3,0/4 3-1-0 (1110) 75% +191
Leela Chess Zero Gen 8 x64 - EtherealRandom (8.97) x64 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 8 x64 - EtherTrueRand 9.21 x64 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 8 x64 - Hanzo the Razor x32 : 1,0/4 0-2-2 (=00=) 25% -191
Leela Chess Zero Gen 8 x64 - Iota 1.0 x32 : 0,5/4 0-3-1 (00=0) 13% -330
Leela Chess Zero Gen 8 x64 - LaMoSca v0.10 x32 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 8 x64 - MFChess 1.3 x32 : 1,0/4 0-2-2 (0=0=) 25% -191
Leela Chess Zero Gen 8 x64 - N.E.G. 1.2 x32 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 8 x64 - NSVChess 0.14 x32 : 1,5/4 1-2-1 (01=0) 38% -85
Leela Chess Zero Gen 8 x64 - POS v1.20 x32 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 8 x64 - Pyotr Amateur Edition v0.6 x32 : 2,0/4 0-0-4 (====) 50% ±0
Leela Chess Zero Gen 8 x64 - Pyotr Novice Edition v2.6 x32 : 3,5/4 3-0-1 (11=1) 88% +346
Leela Chess Zero Gen 8 x64 - Ram 2.0 x32 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 8 x64 - Talvmenni 0.1 x32 : 0,0/4 0-4-0 (0000) 0% -1200
Leela Chess Zero Gen 8 x64 - Teki Random Mover x64 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 8 x64 - Usurpator II x32 : 0,5/4 0-3-1 (00=0) 13% -330
Leela Chess Zero Gen 8 x64 - Xadreco 5.83 x32 : 0,0/4 0-4-0 (0000) 0% -1200
Leela Chess Zero Gen 8 x64 - Youk V1.05 x32 : 1,5/4 1-2-1 (=001) 38% -85
Leela Chess Zero Gen 8 x64 - Zoe 0.1 x32 : 2,0/4 1-1-2 (0==1) 50% ±0
And just for comparison, here's Gen 6
-----------------Leela Chess Zero Gen 6 x64-----------------
Leela Chess Zero Gen 6 x64 - Acqua ver. 20160918 x32 : 2,0/4 2-2-0 (0011) 50% ±0
Leela Chess Zero Gen 6 x64 - BRAMA 05/12/2004 x32 : 0,0/4 0-4-0 (0000) 0% -1200
Leela Chess Zero Gen 6 x64 - CPP1 0.1038 x32 : 3,0/4 3-1-0 (1110) 75% +191
Leela Chess Zero Gen 6 x64 - Dikabi v0.4209 x32 : 2,0/4 0-0-4 (====) 50% ±0
Leela Chess Zero Gen 6 x64 - Easy Peasy 1.0 x32 : 1,0/4 1-3-0 (0010) 25% -191
Leela Chess Zero Gen 6 x64 - EtherealRandom (8.97) x64 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 6 x64 - EtherTrueRand 9.21 x64 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 6 x64 - Hanzo the Razor x32 : 1,0/4 0-2-2 (0==0) 25% -191
Leela Chess Zero Gen 6 x64 - Iota 1.0 x32 : 0,5/4 0-3-1 (00=0) 13% -330
Leela Chess Zero Gen 6 x64 - LaMoSca v0.10 x32 : 3,0/4 2-0-2 (==11) 75% +191
Leela Chess Zero Gen 6 x64 - MFChess 1.3 x32 : 0,5/4 0-3-1 (0=00) 13% -330
Leela Chess Zero Gen 6 x64 - N.E.G. 1.2 x32 : 2,0/4 1-1-2 (10==) 50% ±0
Leela Chess Zero Gen 6 x64 - NSVChess 0.14 x32 : 2,0/4 1-1-2 (0==1) 50% ±0
Leela Chess Zero Gen 6 x64 - POS v1.20 x32 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 6 x64 - Pyotr Amateur Edition v0.6 x32 : 0,5/4 0-3-1 (0=00) 13% -330
Leela Chess Zero Gen 6 x64 - Pyotr Novice Edition v2.6 x32 : 2,0/4 1-1-2 (1==0) 50% ±0
Leela Chess Zero Gen 6 x64 - Ram 2.0 x32 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 6 x64 - Talvmenni 0.1 x32 : 0,0/4 0-4-0 (0000) 0% -1200
Leela Chess Zero Gen 6 x64 - Teki Random Mover x64 : 4,0/4 4-0-0 (1111) 100% +1200
Leela Chess Zero Gen 6 x64 - Usurpator II x32 : 0,0/4 0-4-0 (0000) 0% -1200
Leela Chess Zero Gen 6 x64 - Xadreco 5.83 x32 : 0,0/4 0-4-0 (0000) 0% -1200
Leela Chess Zero Gen 6 x64 - Youk V1.05 x32 : 0,0/4 0-4-0 (0000) 0% -1200
Leela Chess Zero Gen 6 x64 - Zoe 0.1 x32 : 0,5/4 0-3-1 (0=00) 13% -330
The improvement is pretty evident.
Here's the rating list as of now:
163 Usurpator II x32 : 1038.3 107 74 12 21 75 11 692.7 34 30.1
164 Safrad 2.1.35.210 x32 : 999.9 208 112 23 73 59 11 818.4 35 27.9
165 Hanzo the Razor x32 : 985.9 96 52 42 2 76 44 636.7 24 24.0
166 MFChess 1.3 x32 : 934.2 96 54 30 12 72 31 638.8 24 24.0
167 Youk V1.05 x32 : 922.0 154 63 22 69 48 14 932.5 46 44.4
168 StrategicDeep 1.25 x32 : 908.1 58 4 3 51 9 5 1424.0 23 22.5
169 Hippocampe v0.4.2 x32 : 898.5 150 98 18 34 71 12 594.0 15 15.0
170 Zoe 0.1 x32 : 824.3 96 46 28 22 63 29 643.4 24 24.0
171 Pyotr Amateur Edition v0.6 x32 : 782.6 96 42 29 25 59 30 645.2 24 24.0
172 Leela Chess Zero Gen 8 x64 : 782.4 92 45 17 30 58 18 647.5 23 23.0
173 NSVChess 0.14 x32 : 769.6 246 116 70 60 61 28 619.4 30 25.9
174 Dikabi v0.4209 x32 : 734.8 96 23 59 14 55 61 647.2 24 24.0
175 Easy Peasy 1.0 x32 : 669.9 246 118 30 98 54 12 625.1 30 25.9
176 Pyotr Novice Edition v2.6 x32 : 649.2 96 35 21 40 47 22 650.7 24 24.0
177 Leela Chess Zero Gen 6 x64 : 592.0 92 31 18 43 43 20 647.5 23 23.0
178 N.E.G. 1.2 x32 : 510.7 246 89 29 128 42 12 634.1 30 25.9
179 Acqua ver. 20160918 x32 : 510.7 246 95 17 134 42 7 634.1 30 25.9
180 Ram 2.0 x32 : 388.4 246 58 46 142 33 19 641.1 30 25.9
181 Leela Chess Zero Gen 4 x64 : 369.9 150 43 18 89 35 12 629.2 15 15.0
182 CPP1 0.1038 x32 : 323.7 246 45 49 152 28 20 644.8 30 25.9
183 LaMoSca v0.10 x32 : 253.2 246 2 111 133 23 45 648.8 30 25.9
184 POS v1.20 x32 : 144.0 246 18 45 183 16 18 655.0 30 25.9
185 EtherealRandom (8.97) x64 : 51.9 96 2 15 79 10 16 675.6 24 24.0
186 EtherTrueRand 9.21 x64 : 34.8 246 2 48 196 11 20 661.2 30 25.9
187 Teki Random Mover x64 : 0.0 246 0 44 202 9 18 663.2 30 25.9
+190 from Gen 6 to Gen 8, i think it's pretty good.
CMCanavessi
on 16 Mar 2018
Just finished the next match against Stockfish Level 0:
Score of lc_gen9 vs sf_lv0: 42 - 57 - 1 [0.425] 100
Elo difference: -52.51 +/- 69.39
This is an improvement of 59 Elo compared to gen8 (https://github.com/glinscott/leela-chess/issues/100#issuecomment-373554840) and 148 Elo compared to gen7 (https://github.com/glinscott/leela-chess/issues/100#issuecomment-372963434), using Stockfish Level 0 as a metric. So there is a steady improvement, just at a rate less than the self-play Elo which is to be expected.
jkiliani
on 17 Mar 2018
Interestingly for me Leela Gen 9 has no problem beating level 0 stockfish, what settings are you using for that?
JackThomson2
on 17 Mar 2018
800 playouts, and Dirichlet noise. I know it can beat SF Level 0 with more playouts, but I want to keep the metric constant.
jkiliani
on 17 Mar 2018
I think the tests we do should be without noise. Noise is good for self-training, cause it may lead to a new, better move that it will learn from, but for tournaments and elo testing, we should disable noise imho, we want the strongest version of the engine playing those games.
CMCanavessi
on 17 Mar 2018
Yes but the ELO shown in the main page of http://lczero.org/ must use thoses 800 rollouts.. So if we want to compare this graph with "real" ELO we need it to be in the same conditions.
Uriopass
on 17 Mar 2018
@CMCanavessi The problem with not using noise currently is determinism. Until LCZero has random symmetries applied for every neural net evaluation, it will currently play deterministically if you use neither Dirichlet noise nor temperature=1, i.e. proportional move selection. On some systems OpenCL errors remove the deterministic behaviour, but on mine it doesn't since I use CPU. You can easily test that Dirichlet noise affects playing strength only in a very minor way (it might not even do so at all yet since policy priors are still weak), while temperature=1 vastly lowers playing strength.
So until we have a better way to ensure variance in games played (I also opened https://github.com/glinscott/leela-chess/issues/67 for this purpose), keeping Dirichlet noise on always is our best bet.
If you want to see for yourself, just do a cutechess-cli match of two LCZero nets against each other, without OpenCL. They will repeat the same two games over and over.
jkiliani
on 17 Mar 2018
I ran some noise testing earlier today, and it doesn't seem to affect strength too much. (1k playouts)
Score of LeelaChess gen9 1k vs LeelaChess gen9 1k noise: 46 - 36 - 18 [0.555]
Elo difference: 34.86 +/- 62.50
100 of 100 games finished.
tooweaktooslow
on 17 Mar 2018
I had tested the engine with noise against itself with no noise for a very early net, and found also no effect then, but I repeated the experiment on your results. Mine look very similar:
Score of lc_gen9 vs lc_gen9n: 49 - 38 - 13 [0.555] 100
Elo difference: 38.37 +/- 64.51
It's a pity there's no reliable way to enforce variation without weakening the engine... I could probably get away with not using noise against Stockfish, but any match between lczero with different nets would still require it. Maybe once symmetries are implemented, we can retire using Dirichlet noise for evaluation matches.
jkiliani
on 17 Mar 2018
When matching the new net against Stockfish (Lv 0), I didn't find a regression but a very slight improvement compared to gen9:
Score of lc_gen10 vs sf_lv0: 44 - 56 - 0 [0.440] 100
Elo difference: -41.89 +/- 69.44
@Error323 What was the actual match result of gen10 vs gen9?
jkiliani
on 19 Mar 2018
Score of lc_gen10 vs lc_gen9: 43 - 53 - 4 [0.450] 100
Elo difference: -34.86 +/- 67.82
Finished match
Only difference is V2 samples are in the mix. And they have been verified thoroughly, BUT the movecount only goes up to 255 as it's now an unsigned int8.
Error323
on 19 Mar 2018
I'm about to start the usual gauntlet that I run vs 23 other engines. Will inform results later.
CMCanavessi
on 19 Mar 2018
Either way, if the next net is trained on gen8, gen9, and gen10 games, it would have a large sample of very similar strength training data which should allow it to generalise successfully.
So with V2, any games above 255 ply are adjudicated as draw? Or do they simply keep a move count of 255 at every ply beyond that?
jkiliani
on 19 Mar 2018
They keep the same move count. I think the net should not use it as input really... We have 8 history planes for 3fold and a 50 move counter input for the 50 move rule.
It's only producing noise now and could be the reason for the drop in strength with self. Maybe we should set it to always 0?
Error323
on 19 Mar 2018
I don't see a good reason why not... 3-fold and 50 move counter should be enough. The only possible use I can think of for feeding move count to the net as input is to recognise when games are truncated, but at 450 ply, that happens way too rarely for these adjudicated draws to have any effect on training.
Can anyone else here think of a good reason why the training data needs to include move count?
jkiliani
on 19 Mar 2018
Ok so I didn't test Gen 9, so I'm comparing to Gen 8 but from what I'm seeing right now, Gen 10 is a definite (can't say "big" yet) improvement. It's already getting draws and wins vs engines that it never managed to before. We'll see what the raw numbers say in a while.
CMCanavessi
on 19 Mar 2018
Round 1 of 4 completed:
Gen 8 got 12.5 points out of 23 / 11-3-9 WDL
Gen 10 got 14 points out of 23 (with 2 wins vs engines that hadn't beaten before) / 13-2-8 WDL
Calculated rating so far:
189 Leela Chess Zero Gen 10 x64 : 789.5 23 13 2 8 61 9 652.0 23 23.0
190 Leela Chess Zero Gen 8 x64 : 787.6 92 45 17 30 58 18 652.0 23 23.0
Finally, f393628a becomes the first net to beat Stockfish Level 0 with 800 playouts and noise, and it does so by a significant margin:
Score of lc_f393628a vs sf_lv0: 71 - 28 - 1 [0.715] 100
Elo difference: 159.78 +/- 76.76
The Elo difference to the match with gen9 (5c8d14d5) is actually larger than what the direct match by @Error323 yielded. I think there is a good chance that this net would also do very well in @CMCanavessi's tournament, and it looks like at least tentative evidence that a 200k chunk window works well.
I am planning to continue these matches with Stockfish Level 0 until LCZero manages a 85%-90% winrate, and then switch to Level 5 as reference. I think Level 5 will be a good choice since I earlier tested (https://github.com/glinscott/leela-chess/issues/109#issuecomment-372701765) that the supervised net kbb1-64x6-796000.txt is roughly comparable to SF Level 5-6.
Also, we should have clarity in how we refer to networks: Do we continue to call them genxx by the order in which they were promoted to best network, or by their hash? The latter would allow easier reference to candidate nets that were never promoted, but genxx is more intuitive in a way.
jkiliani
on 20 Mar 2018
@jkiliani Let us switch to hashes. The problem is that for newcomers the generation is written nowhere on the website, which makes it confusing.
kiudee
on 20 Mar 2018
Excellent! And indeed the 200K window is now the standard! I also trained a new version last night afterwards with a 100K window, but the MSE on the testset was much higher, indicating overfitting. So nice work @jkiliani :+1:
It's interesting how it suddenly happened to overfit so badly. Something to think about...
about the networks: I'm now calling them by their sha256sum.
Error323
on 20 Mar 2018
So, the Leela Zero tradition it is. But could you truncate the hash to 8 characters for the purpose of naming the files directly downloaded from http://lczero.org/networks? This should suffice for uniqueness, and anything longer than 8 chars becomes really cumbersome. I think even 6 chars would probably suffice to not confuse networks...
jkiliani
on 20 Mar 2018
Personally I'd also like 6 chars. It's memorizable and probably sufficient. I'll discuss with @glinscott
Error323
on 20 Mar 2018
About the training window, Leela Zero went the opposite direction: In the beginning, we used 500k games since that was the value from the AlphaZero paper, even though a much smaller window would almost certainly have been better in the beginning. Later @gcp reduced it to 250k games when it was becoming obvious that the large window obstructed progress in the beginning. Only now, with a very strong and large network, and slow progress, is enlarging the window being discussed again.
I think the 500k from Deepmind must have been picked mainly for the late training phase.
jkiliani
on 20 Mar 2018
probably sufficient
You can expect a duplicate 6-hex-digits name after a few thousand entries.
sf-x
on 20 Mar 2018
It's interesting how it suddenly happened to overfit so badly. Something to think about...
Maybe not enough variety,76747 of 100000 games (am I right?) are from the same crap parameter set with preference for 1.h4
sf-x
on 20 Mar 2018
Maybe not enough variety, 76747 of 100000 games (am I right?) are from the same crap parameter set with preference for 1.h4
No there's a huge gap between when I start training and when I upload a new version. At the time I started training there were only about 20000 games in that net. (Also because I did various other tests in between).
Error323
on 20 Mar 2018
You can expect a duplicate 6-hex-digits name after a few thousand entries.
I just made a spreadsheet to test this, since there are 2^24 different 6-hex hashes. At 2000 entries, there's a roughly 11.5% chance of a hash collision, which seems perfectly acceptable to me given that we will likely only have a couple hundred networks. By the way we should consider gating at 55% winrate, instead of just 50%, for the matches.
jkiliani
on 20 Mar 2018
There are plenty of online calculators, too:
http://everydayinternetstuff.com/2015/04/hash-collision-probability-calculator/
50% chance of collision with 5000 24-bit hashes
mooskagh
on 20 Mar 2018
Or even sprt like Leela Zero which makes more sense mathematicaly?
Uriopass
on 20 Mar 2018
1)like stockfish; 2)practically, not mathematically ;)
sf-x
on 20 Mar 2018
Good to know, thanks 😄. Writing the spreadsheet wasn't hard though.
For promotions, SPRT also seems very sensible to me, but I think we should only terminate early for failed nets, not for passed ones, since a few more matches can calibrate the progress curve better.
@Error323 If the match system is already online now (including automatically updating the progress chart?), could we test it by improving the statistics of a few older matches, to make the progress curve more accurate?
jkiliani
on 20 Mar 2018
Why don't we replace the "ID" column in http://lczero.org/networks with a "Generation" column? It's much more intuitive, and I've seen users already asking for that. I think it's cooler also. If you are talking about different networks and say "Gen 25 vs Gen 17" you instantly know which ones is more recent, but if you say "b91f353d vs 6d2eaec0" that tells you nothing without looking at the table.
CMCanavessi
on 20 Mar 2018
Or just directly use the ID ? I mean it's only 4-off
Uriopass
on 20 Mar 2018
I think we should only terminate early for failed nets, not for passed ones, since a few more matches can calibrate the progress curve better.
Isn't the testing a bottleneck which consumes valuable resources? Someone else (tm) can run matches to establish exact rate of progress; what matters for this project is "is it sufficiently better"? It's also possible to extract some info from SPRT results; it wildly inaccurate but better than nothing.
@vdbergh once proposed to use a even more efficient test than SPRT but then claimed that (from memory) that 10% efficiency gain is not worth sacrificing the simplicity of SPRT. Possibly he just got tired from wrestling with blockheaded maintainer of Stockfish
sf-x
on 20 Mar 2018
Why don't we replace the "ID" column in http://lczero.org/networks with a "Generation" column? It's much more intuitive, and I've seen users already asking for that. I think it's cooler also. If you are talking about different networks and say "Gen 25 vs Gen 17" you instantly know which ones is more recent, but if you say "b91f353d vs 6d2eaec0" that tells you nothing without looking at the table.
True, I just figured that using the hash would be selfsufficient. I.e. can be recomputed from the actual weights. But using the ID is more intuitive, however it does heavily depend on the server's database.
Error323
on 20 Mar 2018
How about: use the first 6 hash characters until the first hash collision. On the first hash collision throw a celebration party, and start using the first 8 characters. Repeat as necessary.
:)
killerducky
on 20 Mar 2018
+1 ^
People are using as low as 4 digits now
sf-x
on 20 Mar 2018
How about a more easy-to-read name using petname as in minigo? If there is a duplicate, you can just run it once again.
isty2e
on 20 Mar 2018
The match of 6690eb against Stockfish Level 0 just finished:
Score of lc_6690eb vs sf_lv0: 79 - 20 - 1 [0.795] 100
Elo difference: 235.45 +/- 87.18
Another solid improvement, roughly correlates to the self-play match. Next net will still be against SF Lv0 first, until I have a result >85% winrate.
jkiliani
on 20 Mar 2018
I'm starting the usual gauntlet vs 23 opponents in a couple of minutes. I think this will be the last gauntlet with these 23 opponents, as they are already too weak but that's better, I can select some new ones, ranging from 1000 to around 1700 elo, with more familiar names, like TSCP. So depending on how this one goes, next one might be a new group.
CMCanavessi
on 20 Mar 2018
There is hardly any point playing 4 games against a random mover now or lamosa, so you could easily leave out the bottom 5.
jjoshua2
on 20 Mar 2018
Yep, but there's also not much point in playing against 500-600 engines, they all end 4-0 anyways. I might also up the number of games vs each opponent to 8.
CMCanavessi
on 20 Mar 2018
With more games there is some use, maybe one will draw or something... I'm excited to see some matches against TSCP and Robicide, and beat up Hippocampe
jjoshua2
on 20 Mar 2018
Well I just checked the progress and Leela is trouncing everyone. It's beating the hell out of all the 1050+ engines... completely amazing.
CMCanavessi
on 20 Mar 2018
Saruman 2017.08.10 64-bit at 1590 is the lowest rated engine with 2017 or 2018 in the title... And it's open source too! Theres actually quite a few once you get to this level and above.
jjoshua2
on 20 Mar 2018
Beautiful, Xadreco is 1050 in my rating list.
[Event "020 - LCZero Gen 12 Gauntlet"]
[Site "RYZEN"]
[Date "2018.03.20"]
[Round "1"]
[White "Leela Chess Zero Gen 12 x64"]
[Black "Xadreco 5.83 x32"]
[Result "1-0"]
[ECO "D34"]
[Opening "QGD Tarrasch"]
[Time "16:27:38"]
[Variation "7.Bg2 Be7 8.O-O"]
[TimeControl "60+1"]
[Termination "normal"]
[PlyCount "135"]
[WhiteType "program"]
[BlackType "program"]
1. c4 e6 2. Nf3 c5 3. g3 d5 4. cxd5 exd5 5. d4 Nc6 6. Bg2 Nf6 7. O-O Be7 8.
Nc3 c4 9. Ne5 {(9.Ne5 Nxe5 10.dxe5 Ng4 11.Nxd5 Nxe5 12.Nxe7 Qxe7 13.f4 Ng4)
-0.22/19 2} O-O {(9. ... Nxe5 10.dxe5 Ng4 11.Bf4 0-0) +0.30/4 34} 10. Bf4
{(10.Bf4 Nh5 11.Nxd5 Nxf4 12.Nxf4 Nxe5 13.dxe5) -0.04/19 2} Nb4 {(10. ...
Be6) +1.00/4 1} 11. a3 {(11.a3 Nc6 12.h4 Nxe5 13.dxe5 Ng4 14.Nxd5 Qxd5)
+0.72/19 2} Nh5 {(11. ... Nc2) +1.20/4 1} 12. axb4 {(12.axb4 g5 13.e3 Nxf4
14.gxf4 gxf4 15.exf4 Bxb4 16.f5 Bxc3 17.bxc3) +2.37/19 2} Nxf4 {(12. ...
Nxf4) +1.05/5 1} 13. gxf4 {(13.gxf4 f6 14.Bxd5+ Kh8 15.Nxc4 Bxb4 16.Qb3
Bxc3 17.Qxc3) +3.31/19 2} Be6 {(13. ... f6) -0.58/5 1} 14. f5 {(14.f5 Bxf5
15.Nxd5 g6 16.e4 Bxe4 17.Bxe4 f5 18.Bf3) +3.15/19 2} Bxf5 {(14. ... Bxf5)
-0.46/5 1} 15. Nxd5 {(15.Nxd5 g6 16.e4 Be6 17.Nxe7+ Qxe7 18.Nxc4 Bxc4
19.f3) +3.31/20 2} Be6 {(15. ... Bxb4 16.Nxb4 Qg5 17.Ng4 Bxg4) -0.92/4 1}
16. Nf4 {(16.Nf4 g5 17.Nxe6 fxe6 18.Bxb7 Rb8 19.Nc6 Rxb7 20.Nxd8 Rxd8 21.e4
Bxb4 22.h3 c3 23.bxc3) +3.10/19 2} Qb6 {(16. ... Qb6 17.Nxe6 Qxe6 18.Bxb7
Qxe5) -1.31/4 1} 17. Nxe6 {(17.Nxe6 fxe6 18.Nxc4 Qxb4 19.Ra4 Qxc4 20.Rxc4
Rad8 21.d5 exd5) +3.89/19 2} Qxe6 {(17. ... Qxe6) -1.53/5 1} 18. Bxb7
{(18.Bxb7 Rab8 19.Ba6 Bxb4 20.Bxc4 Qxc4 21.Nxc4 h6 22.Ne5 f6) +3.78/19 2}
Rad8 {(18. ... Rad8 19.Qc2 Rxd4 20.Nf3 Qxe2) -0.62/4 1} 19. Rxa7 {(19.Rxa7
Qb6 20.Qa4 Rb8 21.Bd5 c3 22.bxc3 Kh8 23.Nxf7+ Kg8) +5.18/19 2} Bxb4 {(19.
... Bxb4 20.Ra4 Qxe5 21.Rxb4 Qxe2) -0.69/4 1} 20. Nc6 {(20.Nc6 f6 21.Nxd8
Rxd8 22.Qa4 c3 23.bxc3 Bxc3 24.d5 Qxd5) +5.54/19 2} Qg6+ {(20. ... Rxd4)
-2.94/5 1} 21. Kh1 {(21.Kh1 Rd7 22.Qa4 Bd6 23.Ne5 Bxe5 24.Qxd7 Bf6 25.Ra8
Rxa8 26.Bxa8 Bd8 27.Qxd8+) +4.77/19 2} Bc5 {(21. ... Qg2+) -3.25/5 1} 22.
Nxd8 {(22.Nxd8 Bxa7 23.Nc6 c3 24.bxc3 Bb6 25.Ne5 Qh5 26.Bc6) +5.26/19 2}
Bxa7 {(22. ... Bxa7) -3.06/5 1} 23. Nc6 {(23.Nc6 c3 24.bxc3 Rb8 25.Nxb8
Bxb8 26.Ba8 h6 27.Qd2 Kh7 28.f3) +5.11/19 2} Re8 {(23. ... Qg2+) -1.96/5 1}
24. Nxa7 {(24.Nxa7 Qb6 25.Qa4 Qxb7+ 26.Qc6 Qxc6+ 27.Nxc6 Rxe2 28.Ne7+ Kf8
29.Nc6 Rxb2 30.d5) +4.39/19 2} Re7 {(24. ... Qd3) -5.47/5 1} 25. Bf3
{(25.Bf3 Rxa7 26.h4 h6 27.h5 Qf6 28.Be4 Qe7 29.Bc6) +3.68/19 2} Rxa7 {(25.
... Rxa7 26.Rg1) -4.52/5 1} 26. Qc1 {(26.Qc1 Qa6 27.Rd1 f6 28.d5 c3)
+2.87/19 2} Re7 {(26. ... Ra1) -2.92/5 1} 27. Qxc4 {(27.Qxc4 Re6 28.Qc5 h5
29.h4 Rf6 30.d5) +4.73/19 2} Re8 {(27. ... Qc2) -3.35/5 1} 28. h4 {(28.h4
Kf8 29.h5 Qf6 30.Bg4 Qg5 31.Bf3 Qf6 32.Bg4 Rd8) +5.56/19 2} Qf6 {(28. ...
Qf5 29.Bg2 Qg4 30.Bd5 Rxe2) -2.23/4 1} 29. Kg2 {(29.Kg2 Qd6 30.Rd1 Rb8
31.b3 Kf8 32.Rd3) +6.91/19 2} Rb8 {(29. ... Rb8 30.e3 Re8 31.Bd5 Rxe3)
-2.83/4 1} 30. b4 {(30.b4 Qxh4 31.b5 Qf6 32.Bc6 Kf8 33.d5) +7.10/19 2} Qxh4
{(30. ... Rd8 31.e3 Re8 32.Bd5 Rxe3) -2.93/4 1} 31. Qc5 {(31.Qc5 Qd8 32.Ra1
f6 33.Ra7 g6) +7.31/19 2} Rd8 {(31. ... Qh2+) -4.23/5 1} 32. b5 {(32.b5 Qf6
33.b6 Rb8 34.Qc7 Rxb6 35.Qxb6) +8.04/19 2} Qxd4 {(32. ... Qh2+) -3.83/5 1}
33. Qxd4 {(33.Qxd4 Rxd4 34.Ra1 Rd8 35.Ra8 Rxa8 36.Bxa8 Kh8 37.b6 f5 38.b7
Kg8 39.b8Q+) +9.87/18 2} Rxd4 {(33. ... Rxd4 34.b6 Rd6 35.b7 Rd8 36.Rc1
Rd2) -3.37/6 1} 34. Ra1 {(34.Ra1 g6 35.b6 Rb4 36.b7 Rb2 37.Ra7 Rb1 38.b8Q+
Kg7 39.Qxb1 Kf6 40.Qb8) +10.27/19 2} f5 {(34. ... Rd2) -2.37/5 1} 35. Ra8+
{(35.Ra8+ Kf7 36.b6 Rb4 37.b7 Rxb7 38.Bxb7 g6 39.Ra7 Kf6 40.Kf3) +11.09/18
2} Kf7 {(35. ... Rd8) -4.07/6 1} 36. b6 {(36.b6 Rb4 37.b7 Rxb7 38.Bxb7 g6
39.Ra7 Kf6 40.Kf3) +12.07/19 2} Rd6 {(36. ... Rd2 37.Rf8+) -3.77/4 1} 37.
b7 {(37.b7 Rb6 38.b8Q Rxb8 39.Rxb8 Ke7 40.Ra8 Kd7 41.Ra3 Kd6) +12.83/18 2}
Rg6+ {(37. ... Rd2) -3.57/5 1} 38. Kf1 {(38.Kf1 Rb6 39.b8Q Rxb8 40.Rxb8 Ke7
41.Ra8 Kf6 42.Kg2) +13.98/18 2} Rb6 {(38. ... Rg1+) -3.77/5 1} 39. b8=Q
{(39.b8Q Rxb8 40.Rxb8 Ke7 41.Ra8 Kd7 42.Ra3 Kd6) +12.47/18 2} Rxb8 {(39.
... Rxb8 40.Rxb8 Kf6) -8.56/7 1} 40. Rxb8 {(40.Rxb8 g6 41.Ra8 Kf6 42.Bd5
Ke5) +12.28/18 2} Ke6 {(40. ... Kf6) -8.56/6 1} 41. Ra8 {(41.Ra8 Ke5 42.Kg2
Kf4 43.Bd5 Ke5) +12.22/18 2} g5 {(41. ... Ke5) -8.06/6 1} 42. Bb7 {(42.Bb7
Ke5 43.Kg2 Kf4 44.Bc6 g4) +11.78/17 2} Kd6 {(42. ... g4) -7.86/5 1} 43. Kg2
{(43.Kg2 Kc7 44.Ba6 h5 45.Kf3) +12.04/17 2} Kc7 {(43. ... Kc5 44.Rc8+)
-9.06/5 1} 44. Ba6 {(44.Ba6 Kb6 45.Kf3 h6 46.Bc4 Kb7 47.Rh8) +11.33/17 1}
h5 {(44. ... Kd6) -7.46/6 1} 45. Kf3 {(45.Kf3 h4 46.Bb5 h3 47.Kg3 h2
48.Kxh2) +12.37/17 1} Kd6 {(45. ... g4+) -7.46/6 1} 46. Kg3 {(46.Kg3 Ke5
47.Bb5 h4+ 48.Kf3) +12.80/17 1} h4+ {(46. ... Kd5) -7.26/6 1} 47. Kh3
{(47.Kh3 Kc7 48.Bb5 Kb6 49.Bd7 Kc7) +12.88/17 1} Ke5 {(47. ... g4+) -6.26/6
1} 48. Bb7 {(48.Bb7 Kf4 49.Bg2 g4+ 50.Kxh4 g3 51.fxg3+ Ke3 52.Kg5 Kxe2)
+12.54/16 1} Ke6 {(48. ... g4+) -6.66/6 1} 49. Bg2 {(49.Bg2 Ke5 50.f3 Kf4
51.Kh2 g4) +12.83/16 1} Ke5 {(49. ... g4+) -7.06/6 1} 50. f3 {(50.f3 Kf4
51.Kh2 Ke3 52.Kh3 Kxe2) +12.42/16 1} Ke6 {(50. ... g4+) -6.76/6 1} 51. Bh1
{(51.Bh1 Ke5 52.Bg2 Kf4 53.Ra3 Ke5) +12.51/16 1} Ke5 {(51. ... g4+) -6.76/6
1} 52. Bg2 {(52.Bg2 Kf4 53.e4 Ke5 54.exf5 Kxf5 55.Ra4 Ke5) +12.32/16 1} Ke6
{(52. ... g4+) -6.76/6 1} 53. e4 {(53.e4 fxe4 54.fxe4 Ke5 55.Kg4 Kf6 56.Bh3
Ke7 57.Kxg5) +12.35/17 1} fxe4 {(53. ... Ke5) -7.26/7 1} 54. fxe4 {(54.fxe4
Ke5 55.Kg4 Kf6 56.Bh3 Ke5 57.Kxg5 Kxe4 58.Kxh4) +12.63/17 1} Ke7 {(54. ...
Ke5) -7.33/6 1} 55. Kg4 {(55.Kg4 Kf6 56.Bh3 Ke5 57.Kxg5 Kxe4 58.Kxh4 Kf4)
+12.85/17 1} Kf6 {(55. ... h3) -8.03/7 1} 56. Bh3 {(56.Bh3 Ke5 57.Kxg5 Kxe4
58.Kxh4 Kf4 59.Bf1 Ke3) +12.72/16 1} Ke5 {(56. ... Ke6) -7.73/7 1} 57. Ra4
{(57.Ra4 Ke6 58.Kxg5+ Ke5 59.Kxh4 Kf6 60.Kg4) +12.70/16 1} Kf6 {(57. ...
Ke6) -7.53/7 1} 58. Ra5 {(58.Ra5 Ke6 59.Kxg5+ Kd6 60.Kf4 Kc6 61.e5)
+13.15/16 1} Ke6 {(58. ... Ke7) -8.03/7 1} 59. Kxg5+ {(59.Kxg5+ Kd6 60.Kf4
Kc6 61.e5 Kb6 62.Ra8 Kc7) +13.74/16 1} Kd6 {(59. ... Ke7) -9.65/7 1} 60.
Kf4 {(60.Kf4 Kc6 61.e5 Kb6 62.Ra1 Kc5 63.e6) +14.37/16 1} Kc7 {(60. ...
Kc6) -9.55/6 1} 61. e5 {(61.e5 Kb6 62.Ra8 Kb7 63.Rh8 Kc6) +14.79/16 1} Kc6
{(61. ... Kb6) -9.95/6 1} 62. e6 {(62.e6 Kd6 63.Ra6+ Ke7 64.Kf5 Kf8)
+14.62/16 1} Kd6 {(62. ... Kb6) -10.05/6 1} 63. Ra6+ {(63.Ra6+ Ke7 64.Ke5
Ke8 65.Kf6 Kd8) +14.92/16 1} Ke7 {(63. ... Kd5) -9.95/6 1} 64. Ke5 {(64.Ke5
Kf8 65.Kf6 Ke8 66.Ra8+) +14.81/16 1} Kf8 {(64. ... Ke8 65.Ra8+) -12.35/6 1}
65. Kf6 {(65.Kf6 Kg8 66.e7 Kh8 67.e8Q+ Kh7 68.Qf7+ Kh8) +17.52/16 1} Kg8
{(65. ... Ke8) -10.75/7 1} 66. e7 {(66.e7 Kh8 67.e8Q+ Kh7 68.Qd7+ Kg8
69.Qg7+) +18.65/16 1} Kh7 {(66. ... Kh8) -11.25/7 1} 67. e8=Q {(67.e8Q Kh6
68.Qh8+) +26.79/16 1} Kh6 {(67. ... Kh6 68.Qh8+) -M30/6 1} 68. Qh8#
{(68.Qh8+) +36.38/16 1} 1-0
I think I would stop your tournament early after one or two rounds then, and get at least one engine in there that will really beat it, and some around its level.
jjoshua2
on 20 Mar 2018
I'd say go through with it if you have the compute. From my matches against Stockfish I'd estimate lc_6690eb around 1150, so the strongest engines on your current match may still pull of the occasional win or draw at least.
jkiliani
on 20 Mar 2018
It finally lost 2 games, vs Usurpator II and Pyotr Amateur. I'll let the gauntlet finish, shouldn't take long at this time control. Then I'll start a new one with the new pack of engines that I'll use till they are all too weak again.
CMCanavessi
on 20 Mar 2018
Ok so I aborted it after 2 round robins, it was too easy for Leela.
Here's the calculated rating:
172 MSCP 1.4 x32 : 1221.2 60 1 4 55 5 7 1758.4 15 15.0
173 BRAMA 05/12/2004 x32 : 1200.7 166 95 47 24 71 28 883.4 40 39.1
174 Tikov 0.6.3 Rev 2 x32 : 1176.6 64 21 10 33 41 16 1250.0 16 16.0
175 Frank 0.58 x32 : 1121.7 64 11 21 32 34 33 1253.4 16 16.0
176 Talvmenni 0.1 x32 : 1096.1 106 70 30 6 80 28 681.0 27 26.8
177 Iota 1.0 x32 : 1086.8 166 80 44 42 61 27 888.9 40 39.1
178 Usurpator II x32 : 1073.1 166 88 24 54 60 14 889.6 40 39.1
179 Leela Chess Zero Gen 12 x64 : 1072.4 46 34 5 7 79 11 656.4 23 23.0
180 Xadreco 5.83 x32 : 1060.8 194 86 23 85 50 12 1016.8 48 47.5
181 Fimbulwinter v5.05 x32 : 1005.9 60 8 9 43 21 15 1261.0 15 15.0
182 Safrad 2.1.35.210 x32 : 1004.2 242 115 28 99 53 12 898.5 35 30.9
183 Hanzo the Razor x32 : 994.0 102 53 46 3 75 45 662.5 26 25.8
184 MFChess 1.3 x32 : 958.2 102 58 30 14 72 29 663.9 26 25.8
185 Youk V1.05 x32 : 930.8 194 66 28 100 41 14 1022.1 48 47.5
186 StrategicDeep 1.25 x32 : 922.8 92 7 4 81 10 4 1418.4 23 23.0
Gen 10 is 862, Gen 8 is 793, Gen 6 is 598, Gen 4 is 369
Results of the shortened gauntlet:
-----------------Leela Chess Zero Gen 12 x64-----------------
Leela Chess Zero Gen 12 x64 - Acqua ver. 20160918 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - BRAMA 05/12/2004 x32 : 0,5/2 0-1-1 (=0) 25% -191
Leela Chess Zero Gen 12 x64 - CPP1 0.1038 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - Dikabi v0.4209 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - Easy Peasy 1.0 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - EtherealRandom (8.97) x64 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - EtherTrueRand 9.21 x64 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - Hanzo the Razor x32 : 1,0/2 0-0-2 (==) 50% ±0
Leela Chess Zero Gen 12 x64 - Iota 1.0 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - LaMoSca v0.10 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - MFChess 1.3 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - N.E.G. 1.2 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - NSVChess 0.14 x32 : 1,5/2 1-0-1 (1=) 75% +191
Leela Chess Zero Gen 12 x64 - POS v1.20 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - Pyotr Amateur Edition v0.6 x32 : 1,0/2 1-1-0 (01) 50% ±0
Leela Chess Zero Gen 12 x64 - Pyotr Novice Edition v2.6 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - Ram 2.0 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - Talvmenni 0.1 x32 : 0,5/2 0-1-1 (0=) 25% -191
Leela Chess Zero Gen 12 x64 - Teki Random Mover x64 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - Usurpator II x32 : 0,0/2 0-2-0 (00) 0% -1200
Leela Chess Zero Gen 12 x64 - Xadreco 5.83 x32 : 1,0/2 1-1-0 (10) 50% ±0
Leela Chess Zero Gen 12 x64 - Youk V1.05 x32 : 2,0/2 2-0-0 (11) 100% +1200
Leela Chess Zero Gen 12 x64 - Zoe 0.1 x32 : 1,0/2 1-1-0 (10) 50% ±0
Ok, here we go. The real deal, I've made a new gauntlet, 8 rounds vs every engine, 25 opponents in total (200 total games), ranging from 1005 elo (Fimbulwinter) to 1850 elo (Skiull).
You can follow the games live at my twitch channel: https://www.twitch.tv/ccls/
It's not gonna be easy for Gen 12 with this pack of rivals, but that's the idea, to see the progress in a couple of days/weeks.
CMCanavessi
on 20 Mar 2018
Around the middle of the 3rd round, Leela has improved its elo to 1083 and has gotten a couple of nice wins vs 1350+ elo engines (Supra and Sabrina), among others. It's doing better than I had expected to be honest. I don't think this pack of engines will last long.
178 Leela Chess Zero Gen 12 x64 : 1083.6 114 45 10 59 44 9 1108.9 48 47.0
CMCanavessi
on 21 Mar 2018
Well, the new gauntlet finished and it was quite harsh. Leela finished with 37/200, we'll see how much that improves in the upcoming generations.
-----------------Leela Chess Zero Gen 12 x64-----------------
Leela Chess Zero Gen 12 x64 - AdaChess v2.1 (GSEI) x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 12 x64 - Ceibo v0.3.65 x64 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 12 x64 - Dragontooth 0.2 Bahamut x64 : 1,5/8 1-6-1 (010000=0) 19% -252
Leela Chess Zero Gen 12 x64 - Eden 0.0.13 x32 : 1,0/8 1-7-0 (00000001) 13% -330
Leela Chess Zero Gen 12 x64 - Enxadrista 1.0 x32 : 2,0/8 2-6-0 (00101000) 25% -191
Leela Chess Zero Gen 12 x64 - Fimbulwinter v5.05 x32 : 5,0/8 5-3-0 (01011011) 63% +92
Leela Chess Zero Gen 12 x64 - Frank 0.58 x32 : 4,0/8 2-2-4 (=100=1==) 50% ±0
Leela Chess Zero Gen 12 x64 - Joanna2002 1.06 x32 : 1,0/8 0-6-2 (00000==0) 13% -330
Leela Chess Zero Gen 12 x64 - KillerQueen 2 beta 3 x32 : 4,0/8 3-3-2 (1010=01=) 50% ±0
Leela Chess Zero Gen 12 x64 - LarsenVB 0.05 x32 : 1,0/8 1-7-0 (00100000) 13% -330
Leela Chess Zero Gen 12 x64 - MSCP 1.4 x32 : 1,0/8 1-7-0 (10000000) 13% -330
Leela Chess Zero Gen 12 x64 - Nanook v0.17 x32 : 2,0/8 0-4-4 (=0==0=00) 25% -191
Leela Chess Zero Gen 12 x64 - Numpty Recharged x64 : 1,5/8 1-6-1 (0=000001) 19% -252
Leela Chess Zero Gen 12 x64 - Pierre v1.7 x32 : 0,5/8 0-7-1 (00000=00) 6% -478
Leela Chess Zero Gen 12 x64 - Piranha 0.5 x32 : 1,0/8 1-7-0 (00000001) 13% -330
Leela Chess Zero Gen 12 x64 - Pulse 1.6.1 x64 : 0,5/8 0-7-1 (000000=0) 6% -478
Leela Chess Zero Gen 12 x64 - Pwned v1.3 x64 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 12 x64 - Sabrina 3.1.25 x64 : 2,5/8 2-5-1 (01000=10) 31% -139
Leela Chess Zero Gen 12 x64 - Satana 2.4.20 x64 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 12 x64 - Simon v1.2 x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 12 x64 - Skiull 0.3 x64 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 12 x64 - Supra 26.0 Pro x64 : 2,5/8 2-5-1 (=0100001) 31% -139
Leela Chess Zero Gen 12 x64 - Tikov 0.6.3 Rev 2 x32 : 3,5/8 3-4-1 (000111=0) 44% -42
Leela Chess Zero Gen 12 x64 - Toledo Nanochess Jan/11/2010 x32 : 2,5/8 2-5-1 (0110=000) 31% -139
Leela Chess Zero Gen 12 x64 - TSCP 1.81 x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Elo now is almost 1100... a huge improvement over Gen 10.
186 Leela Chess Zero Gen 12 x64 : 1097.6 246 61 25 160 30 10 1298.7 48 39.8
198 Leela Chess Zero Gen 10 x64 : 862.1 92 53 11 28 64 12 656.1 23 23.0
201 Leela Chess Zero Gen 8 x64 : 793.3 92 45 17 30 58 18 656.1 23 23.0
206 Leela Chess Zero Gen 6 x64 : 598.5 92 31 18 43 43 20 656.1 23 23.0
210 Leela Chess Zero Gen 4 x64 : 369.6 150 43 18 89 35 12 623.6 15 15.0
Match of gen13 (cd1a1e) against Stockfish Level 0:
Score of lc_cd1a1e vs sf_lv0: 80 - 19 - 1 [0.805] 100
Elo difference: 246.30 +/- 89.11
Only a marginal improvement compared to gen12. I'm going to just give the match against Stockfish Level 5 a shot now just to see what happens. Not using FPU reduction (https://github.com/glinscott/leela-chess/issues/160) for this match although it would certainly give LCZero a big boost against Stockfish, but it's experimental and not agreed on yet by this community.
Results against Stockfish Level 5:
Score of lc_cd1a1e vs sf_lv5: 14 - 84 - 2 [0.150] 100
Elo difference: -301.33 +/- 99.69
Matching SF Lv0 against SF Lv5 directly gave a rating difference of 672 Elo, but the transferred results against LCZero imply a rating difference of ~550 Elo. I tend to believe this more than the Stockfish self-play rating. Since @CMCanavessi's tournament implies that SF Level 0 is ~900 Elo strong, Stockfish Level 5 should be something like 1450 Elo, to use as an anchor for the near future.
I'll do gen14 (209032) after all so I have the comparison with @CMCanavessi's Elo rating.
jkiliani
on 21 Mar 2018
Gen 14 is already out BTW.
CMCanavessi
on 21 Mar 2018
Just started the gauntlet that Gen 12 played against, 25 engines ranging from 1000 to 1850 elo. We'll see what's the real improvement of Gen 14. If it can get to around 50/200 I'll call it a huge success (Gen 12 got 37/200).
I'll update later.
CMCanavessi
on 21 Mar 2018
Gen 14 209032b7 just drew with TSCP at 40/5'
Now that's real progress in such a short time
(I adjudicated a draw rather than wait out the threefold rep.)
```[Event "My Tournament"]
[Site "?"]
[Date "2018.03.21"]
[Round "1"]
[White "leela-wrapper"]
[Black "TSCP"]
[Result "1/2-1/2"]
[ECO "B00"]
[GameEndTime "2018-03-21T22:20:54.402 GMT"]
[GameStartTime "2018-03-21T22:08:34.614 GMT"]
[Opening "King's pawn Opening"]
[PlyCount "98"]
[Termination "adjudication"]
[TimeControl "40/300"]
- e4 {-0.11/20 7.0s} d6 2. d4 {+0.22/20 7.0s} g6 3. Nf3 {+0.44/20 7.0s}
Bg4 {-0.78/7 10.0s} 4. Be2 {+1.23/20 7.0s} Nf6 {-0.68/7 9.7s} - h3 {+1.42/20 7.1s} Bxf3 {-0.52/7 9.3s} 6. Bxf3 {+0.88/20 7.1s}
e5 {-0.45/7 9.0s} 7. dxe5 {+1.72/20 7.1s} dxe5 {-0.51/7 8.7s} - Qxd8+ {+2.09/20 7.1s} Kxd8 {-0.31/7 8.4s} 9. Bd2 {+2.21/21 7.2s}
Kc8 {-0.10/7 8.2s} 10. b4 {+1.97/20 7.2s} Nc6 {+0.35/7 7.9s} - c3 {+1.77/20 7.2s} h5 {+0.28/7 7.6s} 12. Ke2 {+1.45/20 7.3s}
Be7 {+0.49/6 7.4s} 13. b5 {+2.33/20 7.3s} Nd8 {+0.53/7 7.1s} - c4 {+1.58/20 7.3s} Ne6 {+0.70/7 6.9s} 15. Nc3 {+1.81/20 7.4s}
Nd4+ {+0.85/6 6.7s} 16. Kd3 {+0.85/20 7.4s} Rd8 {+1.02/6 6.4s} - Nd5 {+0.79/20 7.4s} Nxd5 {+1.08/7 6.2s} 18. cxd5 {+1.04/21 7.5s}
Nxb5 {+1.42/6 6.0s} 19. Be3 {+1.97/20 7.5s} Bf6 {+1.33/6 5.8s} - a4 {+2.45/20 7.6s} Nd4 {+1.46/7 5.6s} 21. h4 {+1.15/20 7.6s}
Nxf3 {+1.67/7 5.4s} 22. gxf3 {+1.05/20 7.6s} c6 {+1.73/7 5.3s} - Rac1 {+0.92/21 7.7s} Kb8 {+1.63/7 5.1s} 24. f4 {+0.45/20 7.7s}
cxd5 {+2.46/7 4.9s} 25. fxe5 {+0.06/20 7.8s} Bxe5 {+2.07/7 4.7s} - Rhg1 {+0.23/21 7.8s} d4 {+2.78/7 4.6s} 27. Bg5 {+0.66/20 7.8s}
Re8 {+2.24/7 4.4s} 28. f4 {+0.91/20 7.8s} Bc7 {+1.41/7 4.3s} - f5 {+0.23/21 7.8s} gxf5 {+2.06/8 4.1s} 30. exf5 {+0.80/20 7.8s}
b5 {+1.44/7 4.0s} 31. axb5 {+1.50/20 7.8s} Be5 {+0.78/7 3.9s} - Rc6 {+2.55/20 7.8s} Kb7 {+0.45/7 3.7s} 33. Bf6 {+1.49/20 7.8s}
Rab8 {-0.35/7 3.6s} 34. Rg7 {+3.59/20 7.7s} Bxf6 {-0.77/8 3.5s} - Rxf6 {+4.55/20 7.6s} Kc8 {-0.97/8 3.4s} 36. Rfxf7 {+5.57/20 7.6s}
Rxb5 {-0.90/8 3.3s} 37. f6 {+5.13/21 7.6s} Rb4 {-0.88/7 3.2s} - Rh7 {+5.33/20 7.6s} Re3+ {-0.44/7 3.1s} 39. Kd2 {+4.25/20 7.8s}
Rb2+ {-0.60/7 3.0s} 40. Kd1 {+3.60/21 8.4s} Rbe2 {-0.60/6 2.9s} - Rfg7 {+4.40/21 15s} Re1+ {0.00/8 13s} 42. Kc2 {+4.39/23 14s}
R1e2+ {0.00/7 12s} 43. Kc1 {+4.37/23 14s} Re1+ {0.00/7 12s} - Kd2 {+4.31/23 13s} R1e2+ {0.00/8 12s} 45. Kc1 {+4.17/24 13s}
Re1+ {0.00/8 11s} 46. Kd2 {+3.98/23 12s} R1e2+ {0.00/8 11s} - Kd1 {+4.00/23 12s} Re1+ {0.00/8 10s} 48. Kc2 {+3.83/24 11s}
R1e2+ {0.00/7 10s} 49. Kb1 {+2.36/25 11s}
Re1+ {0.00/7 9.7s, Draw by adjudication: user decision} 1/2-1/2
```
gsobala
on 21 Mar 2018
It just beat Pwned 1.3 in my gauntlet, which is almost exactly as strong as TSCP. TC is 1 min + 1 sec, so even shorter. Looking great!
136 Pwned v1.3 x64 : 1794.8 254 92 19 143 40 7 2015.2 85 73.4
137 TSCP 1.81 x32 : 1792.2 254 89 24 141 40 9 2015.3 85 73.4
Gen 14 (209032) finished its match against Stockfish Level 5:
Score of lc_209032 vs sf_lv5: 17 - 81 - 2 [0.180] 100
Elo difference: -263.42 +/- 91.50
The improvement fits the self-play match well. I'll use @CMCanavessi's gauntlet of Gen 14 to check whether my estimate of 1450 Elo for SF Lv 5 was plausible.
jkiliani
on 22 Mar 2018
This is really fun to read. Our baby grows up so fast :')
Error323
on 22 Mar 2018
The gauntlet for Gen 14 finished. It got 52/200 (Gen 12 had gotten 37/200). A nice improvement.
-----------------Leela Chess Zero Gen 14 x64-----------------
Leela Chess Zero Gen 14 x64 - AdaChess v2.1 (GSEI) x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 14 x64 - Ceibo v0.3.65 x64 : 0,5/8 0-7-1 (000000=0) 6% -478
Leela Chess Zero Gen 14 x64 - Dragontooth 0.2 Bahamut x64 : 4,5/8 4-3-1 (=1001110) 56% +42
Leela Chess Zero Gen 14 x64 - Eden 0.0.13 x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 14 x64 - Enxadrista 1.0 x32 : 6,0/8 6-2-0 (11011110) 75% +191
Leela Chess Zero Gen 14 x64 - Fimbulwinter v5.05 x32 : 7,0/8 7-1-0 (11011111) 88% +346
Leela Chess Zero Gen 14 x64 - Frank 0.58 x32 : 5,0/8 4-2-2 (1110==10) 63% +92
Leela Chess Zero Gen 14 x64 - Joanna2002 1.06 x32 : 1,0/8 0-6-2 (0=0=0000) 13% -330
Leela Chess Zero Gen 14 x64 - KillerQueen 2 beta 3 x32 : 4,5/8 4-3-1 (11001=01) 56% +42
Leela Chess Zero Gen 14 x64 - LarsenVB 0.05 x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 14 x64 - MSCP 1.4 x32 : 1,5/8 1-6-1 (1=000000) 19% -252
Leela Chess Zero Gen 14 x64 - Nanook v0.17 x32 : 3,5/8 1-2-5 (==10=0==) 44% -42
Leela Chess Zero Gen 14 x64 - Numpty Recharged x64 : 1,5/8 1-6-1 (00100=00) 19% -252
Leela Chess Zero Gen 14 x64 - Pierre v1.7 x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 14 x64 - Piranha 0.5 x32 : 1,0/8 1-7-0 (00000100) 13% -330
Leela Chess Zero Gen 14 x64 - Pulse 1.6.1 x64 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 14 x64 - Pwned v1.3 x64 : 1,0/8 1-7-0 (10000000) 13% -330
Leela Chess Zero Gen 14 x64 - Sabrina 3.1.25 x64 : 2,0/8 1-5-2 (00=00=01) 25% -191
Leela Chess Zero Gen 14 x64 - Satana 2.4.20 x64 : 2,5/8 2-5-1 (10000=10) 31% -139
Leela Chess Zero Gen 14 x64 - Simon v1.2 x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 14 x64 - Skiull 0.3 x64 : 0,5/8 0-7-1 (00000=00) 6% -478
Leela Chess Zero Gen 14 x64 - Supra 26.0 Pro x64 : 1,5/8 1-6-1 (=1000000) 19% -252
Leela Chess Zero Gen 14 x64 - Tikov 0.6.3 Rev 2 x32 : 5,0/8 5-3-0 (10101110) 63% +92
Leela Chess Zero Gen 14 x64 - Toledo Nanochess Jan/11/2010 x32 : 3,5/8 3-4-1 (=1001010) 44% -42
Leela Chess Zero Gen 14 x64 - TSCP 1.81 x32 : 0,0/8 0-8-0 (00000000) 0% -1200
And the current rating list:
183 Leela Chess Zero Gen 14 x64 : 1200.5 200 42 20 138 26 10 1452.0 25 25.0
184 BRAMA 05/12/2004 x32 : 1196.0 166 95 47 24 71 28 880.8 40 39.1
185 Tikov 0.6.3 Rev 2 x32 : 1160.1 80 28 11 41 42 14 1224.9 18 17.4
186 Frank 0.58 x32 : 1111.6 80 15 27 38 36 34 1227.4 18 17.4
187 Leela Chess Zero Gen 12 x64 : 1098.5 246 61 25 160 30 10 1303.1 48 39.8
188 Talvmenni 0.1 x32 : 1094.6 106 70 30 6 80 28 680.1 27 26.8
189 Iota 1.0 x32 : 1082.5 166 80 44 42 61 27 886.2 40 39.1
190 Usurpator II x32 : 1069.0 166 88 24 54 60 14 886.9 40 39.1
191 Xadreco 5.83 x32 : 1061.2 194 86 23 85 50 12 1017.6 48 47.5
192 Safrad 2.1.35.210 x32 : 1004.4 242 115 28 99 53 12 899.1 35 30.9
193 Hanzo the Razor x32 : 993.8 102 53 46 3 75 45 662.3 26 25.8
194 Fimbulwinter v5.05 x32 : 987.4 76 12 9 55 22 12 1233.5 17 16.4
195 MFChess 1.3 x32 : 958.0 102 58 30 14 72 29 663.7 26 25.8
196 Youk V1.05 x32 : 931.0 194 66 28 100 41 14 1023.0 48 47.5
197 StrategicDeep 1.25 x32 : 923.8 92 7 4 81 10 4 1420.5 23 23.0
198 Hippocampe v0.4.2 x32 : 894.1 150 98 18 34 71 12 588.8 15 15.0
199 Leela Chess Zero Gen 10 x64 : 861.6 92 53 11 28 64 12 655.7 23 23.0
200 Zoe 0.1 x32 : 827.4 102 47 29 26 60 28 668.9 26 25.8
201 Pyotr Amateur Edition v0.6 x32 : 821.8 102 46 30 26 60 29 669.1 26 25.8
202 Leela Chess Zero Gen 8 x64 : 792.9 92 45 17 30 58 18 655.7 23 23.0
203 NSVChess 0.14 x32 : 777.0 252 118 71 63 61 28 626.7 32 27.2
204 Dikabi v0.4209 x32 : 737.3 102 23 61 18 52 60 672.4 26 25.8
205 Easy Peasy 1.0 x32 : 666.4 252 118 30 104 53 12 632.8 32 27.2
206 Pyotr Novice Edition v2.6 x32 : 649.6 102 35 22 45 45 22 675.8 26 25.8
207 Leela Chess Zero Gen 6 x64 : 598.3 92 31 18 43 43 20 655.7 23 23.0
208 Acqua ver. 20160918 x32 : 509.3 252 95 17 140 41 7 641.5 32 27.2
209 N.E.G. 1.2 x32 : 509.3 252 89 29 134 41 12 641.5 32 27.2
210 Ram 2.0 x32 : 387.7 252 58 46 148 32 18 648.3 32 27.2
211 Leela Chess Zero Gen 4 x64 : 369.6 150 43 18 89 35 12 623.8 15 15.0
Gen 15 (6a5ccd) against Stockfish Level 5:
Score of lc_6a5ccd vs sf_lv5: 23 - 73 - 4 [0.250] 100
Elo difference: -190.85 +/- 78.70
Solid improvement this time, LZ is fast closing the gap. Glad to hear about the Gen 14 gauntlet results by the way, they seems to indicate that 1450 is quite accurate for Stockfish Lv 5. Once Level 5 is beaten, I'll try a match against the kingbase supervised net.
Almost glad that we have failed nets sometimes, or I couldn't keep up with the Stockfish and the FPU matches anymore 😀
jkiliani
on 22 Mar 2018
After the superfinal of my tournament is finished (should be early tomorrow), I'll make some kind of "special" broadcast with LCZero (probably gen 16 or whatever gen is the latest) with long time controls against TSCP (maybe 100 games match with reversed openings), which is known by every single chess engine programmer out there and is ~1790 elo in my ranking. Should be fun if Leela can beat it or get close to that, and should give us some free promotion.
I might also start talking with the TCEC guys, maybe they want to have LCZ for next season which should be in 3-4 months time, and by then LCZ might be around 2500? That would be a HUGE success for both LCZ and TCEC, after all the A0 hype.
CMCanavessi
on 22 Mar 2018
I think in 3-4 months LCZero will likely be somewhere around 3000 already. With the speed we're currently generating games, I think at that point we'll have upsized the neural net already at least once, probably to 128 filters, 10 blocks since the experience with Leela Zero shows that this is a very effective combination for a good performance net.
jkiliani
on 22 Mar 2018
Well, 3000 would be even better, but I always try to be conservative :)
If we account for TCEC's super long time controls, I think LCZ could surprise a lot of people.
CMCanavessi
on 22 Mar 2018
Agreed, the larger the neural net, the better it will scale relative to Alpha-Beta engines.
jkiliani
on 22 Mar 2018
Gen 16 (98240a) vs SF L5:
Score of lc_98240a vs sf_lv5: 26 - 69 - 5 [0.285] 100
Elo difference: -159.78 +/- 74.73
Gap is diminishing, but will likely still take a few network generations before LCZero is on par. Elo estimation from this match: ~1300.
jkiliani
on 23 Mar 2018
Gen 16 Gauntlet:
-----------------Leela Chess Zero Gen 16 x64-----------------
Leela Chess Zero Gen 16 x64 - AdaChess v2.1 (GSEI) x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 16 x64 - Ceibo v0.3.65 x64 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 16 x64 - Dragontooth 0.2 Bahamut x64 : 3,5/8 3-4-1 (01011=00) 44% -42
Leela Chess Zero Gen 16 x64 - Eden 0.0.13 x32 : 0,5/8 0-7-1 (0=000000) 6% -478
Leela Chess Zero Gen 16 x64 - Enxadrista 1.0 x32 : 4,5/8 3-2-3 (==10=110) 56% +42
Leela Chess Zero Gen 16 x64 - Fimbulwinter v5.05 x32 : 5,0/8 5-3-0 (10011011) 63% +92
Leela Chess Zero Gen 16 x64 - Frank 0.58 x32 : 3,5/8 3-4-1 (10=10001) 44% -42
Leela Chess Zero Gen 16 x64 - Joanna2002 1.06 x32 : 1,0/8 1-7-0 (00000100) 13% -330
Leela Chess Zero Gen 16 x64 - KillerQueen 2 beta 3 x32 : 6,5/8 5-0-3 (111=1==1) 81% +252
Leela Chess Zero Gen 16 x64 - LarsenVB 0.05 x32 : 1,0/8 1-7-0 (00001000) 13% -330
Leela Chess Zero Gen 16 x64 - MSCP 1.4 x32 : 4,5/8 4-3-1 (1=001101) 56% +42
Leela Chess Zero Gen 16 x64 - Nanook v0.17 x32 : 5,0/8 3-1-4 (1=01=1==) 63% +92
Leela Chess Zero Gen 16 x64 - Numpty Recharged x64 : 1,5/8 1-6-1 (0100=000) 19% -252
Leela Chess Zero Gen 16 x64 - Pierre v1.7 x32 : 2,5/8 2-5-1 (=0100010) 31% -139
Leela Chess Zero Gen 16 x64 - Piranha 0.5 x32 : 1,0/8 1-7-0 (00001000) 13% -330
Leela Chess Zero Gen 16 x64 - Pulse 1.6.1 x64 : 1,0/8 1-7-0 (10000000) 13% -330
Leela Chess Zero Gen 16 x64 - Pwned v1.3 x64 : 1,0/8 1-7-0 (01000000) 13% -330
Leela Chess Zero Gen 16 x64 - Sabrina 3.1.25 x64 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 16 x64 - Satana 2.4.20 x64 : 2,5/8 2-5-1 (100100=0) 31% -139
Leela Chess Zero Gen 16 x64 - Simon v1.2 x32 : 1,5/8 1-6-1 (000100=0) 19% -252
Leela Chess Zero Gen 16 x64 - Skiull 0.3 x64 : 2,5/8 2-5-1 (01000=01) 31% -139
Leela Chess Zero Gen 16 x64 - Supra 26.0 Pro x64 : 3,5/8 3-4-1 (0=110100) 44% -42
Leela Chess Zero Gen 16 x64 - Tikov 0.6.3 Rev 2 x32 : 5,5/8 5-2-1 (0111101=) 69% +139
Leela Chess Zero Gen 16 x64 - Toledo Nanochess Jan/11/2010 x32 : 3,5/8 2-3-3 (0011===0) 44% -42
Leela Chess Zero Gen 16 x64 - TSCP 1.81 x32 : 0,5/8 0-7-1 (000000=0) 6% -478
Current ratings:
179 Leela Chess Zero Gen 16 x64 : 1245.8 200 49 25 126 31 13 1442.6 25 25.0
185 Leela Chess Zero Gen 14 x64 : 1192.9 200 42 20 138 26 10 1442.6 25 25.0
188 Leela Chess Zero Gen 12 x64 : 1099.6 250 64 26 160 31 10 1291.3 49 40.8
201 Leela Chess Zero Gen 10 x64 : 861.1 92 53 11 28 64 12 655.3 23 23.0
204 Leela Chess Zero Gen 8 x64 : 792.5 92 45 17 30 58 18 655.3 23 23.0
209 Leela Chess Zero Gen 6 x64 : 598.1 92 31 18 43 43 20 655.3 23 23.0
213 Leela Chess Zero Gen 4 x64 : 369.6 150 43 18 89 35 12 624.0 15 15.0
Gen 12 finished with 37/200
Gen 14 finished with 52/200
Gen 16 finished with 61.5/200
CMCanavessi
on 23 Mar 2018
Further progress, gen 17 (38576a) against Stockfish Level 5:
Score of lc_38576a vs sf_lv5: 33 - 61 - 6 [0.360] 100
Elo difference: -99.95 +/- 69.62
The gap is closing, maybe two more nets...
jkiliani
on 23 Mar 2018
Gen 18 (8c1c61) vs SF Lv5:
Score of lc_8c1c61 vs sf_lv5: 39 - 56 - 5 [0.415] 100
Elo difference: -59.64 +/- 68.14
One more net is probably optimistic, I doubt 7428c7 could do it based on Elo Delta. But shouldn't be long now.
jkiliani
on 24 Mar 2018
And surprisingly, Id29 (9fa03e) did actually beat Stockfish Level 5, although narrowly:
Score of lc_id29 vs sf_lv5: 51 - 44 - 5 [0.535] 100
Elo difference: 24.36 +/- 67.27
From the Self-play Elo progression I did not expect this yet, but it appears that the strengths and weaknesses of Leela Chess are currently shifting in a way that roughly balances out against itself, but helps against Stockfish. Or it may simply be statistical noise since I'm only doing 100 games each.
Very soon it may be time to test the reinforcement learning nets against the Kingbase supervised net.
After LCZero can beat SF Lv 5 with 85% winrate, I will start pitting it against Stockfish Level 10. From my tests between SF Lv5 and SF Lv10, the rating difference between the two should be around 550 Elo, similar to Lv0 to Lv5, and Level 10 would be very roughly in the 2000 Elo range.
I am now switching to network Id numbers by the way for my match reports, since "generation" is not an official designation and also has very limited usefulness with a lot of failed nets and narrow passes in between.
jkiliani
on 24 Mar 2018
Gen 20 gauntlet:
-----------------Leela Chess Zero Gen 20 x64-----------------
Leela Chess Zero Gen 20 x64 - AdaChess v2.1 (GSEI) x32 : 2,0/8 2-6-0 (01001000) 25% -191
Leela Chess Zero Gen 20 x64 - Ceibo v0.3.65 x64 : 1,0/8 1-7-0 (00001000) 13% -330
Leela Chess Zero Gen 20 x64 - Dragontooth 0.2 Bahamut x64 : 6,0/8 5-1-2 (=10=1111) 75% +191
Leela Chess Zero Gen 20 x64 - Eden 0.0.13 x32 : 1,0/8 1-7-0 (00000001) 13% -330
Leela Chess Zero Gen 20 x64 - Enxadrista 1.0 x32 : 3,0/8 3-5-0 (01010010) 38% -85
Leela Chess Zero Gen 20 x64 - Fimbulwinter v5.05 x32 : 8,0/8 8-0-0 (11111111) 100% +1200
Leela Chess Zero Gen 20 x64 - Frank 0.58 x32 : 5,5/8 5-2-1 (111001=1) 69% +139
Leela Chess Zero Gen 20 x64 - Joanna2002 1.06 x32 : 3,5/8 1-2-5 (00===1==) 44% -42
Leela Chess Zero Gen 20 x64 - KillerQueen 2 beta 3 x32 : 4,5/8 4-3-1 (01=11001) 56% +42
Leela Chess Zero Gen 20 x64 - LarsenVB 0.05 x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 20 x64 - MSCP 1.4 x32 : 6,0/8 6-2-0 (01111011) 75% +191
Leela Chess Zero Gen 20 x64 - Nanook v0.17 x32 : 4,5/8 2-1-5 (1=1=0===) 56% +42
Leela Chess Zero Gen 20 x64 - Numpty Recharged x64 : 2,0/8 2-6-0 (00000101) 25% -191
Leela Chess Zero Gen 20 x64 - Pierre v1.7 x32 : 5,0/8 5-3-0 (11011010) 63% +92
Leela Chess Zero Gen 20 x64 - Piranha 0.5 x32 : 2,0/8 2-6-0 (00000110) 25% -191
Leela Chess Zero Gen 20 x64 - Pulse 1.6.1 x64 : 3,0/8 2-4-2 (00101=0=) 38% -85
Leela Chess Zero Gen 20 x64 - Pwned v1.3 x64 : 1,5/8 1-6-1 (1=000000) 19% -252
Leela Chess Zero Gen 20 x64 - Sabrina 3.1.25 x64 : 3,0/8 2-4-2 (00==0011) 38% -85
Leela Chess Zero Gen 20 x64 - Satana 2.4.20 x64 : 2,5/8 2-5-1 (01000=10) 31% -139
Leela Chess Zero Gen 20 x64 - Simon v1.2 x32 : 1,0/8 0-6-2 (00==0000) 13% -330
Leela Chess Zero Gen 20 x64 - Skiull 0.3 x64 : 0,0/8 0-8-0 (00000000) 0% -1200
Leela Chess Zero Gen 20 x64 - Supra 26.0 Pro x64 : 3,0/8 3-5-0 (00001110) 38% -85
Leela Chess Zero Gen 20 x64 - Tikov 0.6.3 Rev 2 x32 : 5,5/8 5-2-1 (1111001=) 69% +139
Leela Chess Zero Gen 20 x64 - Toledo Nanochess Jan/11/2010 x32 : 4,0/8 2-2-4 (1=0=01==) 50% ±0
Leela Chess Zero Gen 20 x64 - TSCP 1.81 x32 : 0,0/8 0-8-0 (00000000) 0% -1200
Gen 16 got 60/200
Gen 20 got 77.5/200
Big improvement over Gen 16
Calculated elo:
Gen 14 | 1191.3
Gen 16 | 1235.8
Gen 20 | 1325.8
Small regression against SF Level 5 with id31 (dd080d):
Score of lc_id31 vs sf_lv5: 43 - 52 - 5 [0.455] 100
Elo difference: -31.35 +/- 67.39
I would attribute this mostly to the probably large error in strength tests at 100 games. Looks like current nets are roughly on par with SF Lv 5.
jkiliani
on 25 Mar 2018
Id 37 wins again versus Stockfish Level 5, although narrowly:
Score of lc_id37 vs sf_lv5: 53 - 44 - 3 [0.545] 100
Elo difference: 31.35 +/- 68.12
And the result for Id 38:
Score of lc_id38 vs sf_lv5: 55 - 42 - 3 [0.565] 100
Elo difference: 45.42 +/- 68.44
We may have quite a bit of rank inflation here. The statistics by @evalon32 (#92) and @Uriopass (#166) clearly show that the nets are still learning, but apparently that does not generally correspond to better performance against a handicapped Stockfish. Let's see what @CMCanavessi's tournament says, but my opinion is shifting toward always-promote now.
jkiliani
on 26 Mar 2018
My gauntlet still has 2 rounds to go (50 games) but I've calculated Gen 24's elo so far and this is the result (might change a bit later)
166 Leela Chess Zero Gen 24 x64 : 1417.5 150 66 14 70 49 9 1433.3 25 25.0
171 Leela Chess Zero Gen 20 x64 : 1322.3 200 64 27 109 39 14 1433.3 25 25.0
184 Leela Chess Zero Gen 16 x64 : 1233.5 200 47 26 127 30 13 1433.3 25 25.0
187 Leela Chess Zero Gen 14 x64 : 1189.4 200 42 20 138 26 10 1433.3 25 25.0
190 Leela Chess Zero Gen 12 x64 : 1097.7 250 64 26 160 31 10 1283.8 49 40.8
203 Leela Chess Zero Gen 10 x64 : 861.2 92 53 11 28 64 12 655.4 23 23.0
206 Leela Chess Zero Gen 8 x64 : 792.6 92 45 17 30 58 18 655.4 23 23.0
211 Leela Chess Zero Gen 6 x64 : 598.1 92 31 18 43 43 20 655.4 23 23.0
215 Leela Chess Zero Gen 4 x64 : 369.4 150 43 18 89 35 12 623.4 15 15.0
Edit: those ratings are with a 1 min + 1 second TC.
Edit2: I might wait for Gen 30 for the next gauntlet, unless there's some kind of huge jump in one of the networks.
CMCanavessi
on 26 Mar 2018
For clarity's sake, could you please also include either network hash or Id when posting gauntlets? I switched to Id along with both @evalon32 and @Uriopass since it's an official designation that can be checked on the site, unlike generation.
jkiliani
on 26 Mar 2018
Gen 24: 75b0a6c1b7c5a48a5d1b8ce52fc7b673adc02508a1dfc8af3ab3a10bff1ba2cb (ID 37)
Gen 20: 9fa03e74ea013cd77bcf8cc82c11543317a76839a912feaf6afb7aa1c7d6f1bc (ID 29)
Gen 16: 98240a5b3d5d27102475173ad02ed0d2cf9db033d0037183cbe4b7631366aeec (ID 22)
CMCanavessi
on 26 Mar 2018
opinion is shifting toward always-promote now.
Remember that now if the progress stalls (50% fail), Error323 tweaks optimization parameters (lowering the step size, for instance).
sf-x
on 27 Mar 2018
Match result of Id 45 (9e5802) against Stockfish Level 5, at 800 playouts:
Score of lc_id45 vs sf_lv5: 64 - 31 - 5 [0.665] 100
Elo difference: 119.11 +/- 71.29
Compared to the match with Id 38, Leela gained ~75 Elo here, which is not much considering a self-play Elo difference of 284. Nonetheless, progress.
jkiliani
on 27 Mar 2018
I'm running the gauntlet of Gen 30 (ID 44) and, even when there are 60 games remaining, we've already broken the 1500 elo barrier in my scale :D
172 Leela Chess Zero Gen 30 (ID 44) x64 : 1502.5 140 70 18 52 56 13 1443.8 25 24.9
179 Leela Chess Zero Gen 24 x64 : 1403.3 200 84 22 94 48 11 1429.3 25 25.0
185 Leela Chess Zero Gen 20 x64 : 1319.9 200 64 27 109 39 14 1429.3 25 25.0
198 Leela Chess Zero Gen 16 x64 : 1231.6 200 47 26 127 30 13 1429.3 25 25.0
200 Leela Chess Zero Gen 14 x64 : 1187.7 200 42 20 138 26 10 1429.3 25 25.0
203 Leela Chess Zero Gen 12 x64 : 1096.6 250 64 26 160 31 10 1280.6 49 40.8
216 Leela Chess Zero Gen 10 x64 : 860.9 92 53 11 28 64 12 655.1 23 23.0
219 Leela Chess Zero Gen 8 x64 : 792.2 92 45 17 30 58 18 655.1 23 23.0
224 Leela Chess Zero Gen 6 x64 : 597.8 92 31 18 43 43 20 655.1 23 23.0
228 Leela Chess Zero Gen 4 x64 : 369.2 150 43 18 89 35 12 623.0 15 15.0
Let's see if it's still +1500 after it ends.
CMCanavessi
on 27 Mar 2018
Gauntlet is finished, here's the progression of the tested networks:
CMCanavessi
on 28 Mar 2018
Result of Id 55 (4b3d91) against Stockfish Level 5:
Score of lc_id55 vs sf_lv5: 80 - 18 - 2 [0.810] 100
Elo difference: 251.89 +/- 89.32
This is a huge step forward indeed with 130 Elo gain compared to https://github.com/glinscott/leela-chess/issues/109#issuecomment-376659819 and supports the very lopsided result @Error323 had when testing the net locally against Id 45. It may actually be time to start testing Leela Chess against Stockfish Level 10 now, but I'm not expecting a lot of wins from that yet.
And here's that result:
Score of lc_id55 vs sf_lv10: 20 - 66 - 14 [0.270] 100
Elo difference: -172.78 +/- 71.04
A surprisingly strong performance against Stockfish Level 10 (still 1 sec / 40 moves), especially since the internal tournament I did earlier indicated a difference of 650 Elo between SF Lv10 and Lv5. This is almost certainly rating inflation, so I'm going to provisionally estimate Stockfish Lv10 at the settings I used as 1900 Elo, which would put Leela in the 1700-1750 range. I'm very interested in @CMCanavessi's next gauntlet to fine tune this estimate.
jkiliani
on 29 Mar 2018
Excellent, it's good to see we're back on track again!
Error323
on 29 Mar 2018
Id 65 against Stockfish level 10:
Score of lc_id65 vs sf_lv10: 24 - 63 - 13 [0.305] 100
Elo difference: -143.07 +/- 69.27
About 30 Elo improvement compared to the last match, with Id 55. Although we're still improving, I think we're getting rather close to the ceiling for the 6x64 architecture.
jkiliani
on 31 Mar 2018
@jkiliani Statistics for castling frequency or king movement seem to show a continuation of improvement and no flattening. Don't you think it's a sign that we are far away from being bounded by the current architecture?
https://github.com/glinscott/leela-chess/issues/166
zz4032
on 31 Mar 2018
So I've changed the usual gauntlet because it was already too weak for Leela. Then I matched LCZero ID 55 with the new pack of opponents and here's the result:
Engine Score Le
01: Leela Chess Zero Gen 33 (ID 55) x64 61.0/200 ········
02: micro-Max 4.8 x32 8.0/8 11111111
02: Embla 1.0.2 x64 8.0/8 11111111
02: Ecce rev. 508 x64 8.0/8 11111111
02: Vice 1.1 x32 8.0/8 11111111
06: Warrior 1.0.3 x32 7.5/8 1111=111
07: Faile 1.4 x32 7.0/8 11111101
07: Ceibo v0.4 x64 7.0/8 11110111
07: Embracer 1.12 x32 7.0/8 11111011
10: Grizzly V1.40.1b x32 6.0/8 10111110
10: SnailChess 4.013 x32 6.0/8 11=01=11
10: Gerbil 02 x64 6.0/8 10111011
10: SamChess x32 6.0/8 10110111
10: Skiull 0.3 x64 6.0/8 111=011=
10: Macromix 2.01 x32 6.0/8 0111=1=1
16: Pigeon 1.5.1 x64 5.5/8 0101111=
17: Laurifer 1.0 x32 4.5/8 011001=1
17: Eia v0.3 x32 4.5/8 1=0101==
19: Robin 0.983 x32 4.0/8 10010==1
19: LarsenVB 0.05 x32 4.0/8 011=010=
19: TSCP 1.81 x32 4.0/8 01001=1=
22: Pwned v1.3 x64 3.5/8 0000111=
22: RataAeroespacial 0.2.1 x32 3.5/8 0110001=
24: HoiChess 0.21.0 x64 3.0/8 01110000
24: Simon v1.2 x32 3.0/8 0=10001=
24: Predateur 2.2.1 x32 3.0/8 00100110
200 games played / Tournament finished
Name of the tournament: 034 - LCZero Gen 33 Gauntlet II
Honestly I was aiming at around 30-35/200, but it scored double that ammount. Here's the current elo list of every version I've tested:
196 Leela Chess Zero Gen 33 (ID 55) x64 : 1667.4 400 185 35 180 51 9 1643.2 45 43.5
219 Leela Chess Zero Gen 30 (ID 44) x64 : 1503.4 200 106 22 72 59 11 1424.1 25 25.0
227 Leela Chess Zero Gen 24 x64 : 1399.2 200 84 22 94 48 11 1424.1 25 25.0
232 Leela Chess Zero Gen 20 x64 : 1316.1 200 64 27 109 39 14 1424.1 25 25.0
245 Leela Chess Zero Gen 16 x64 : 1228.0 200 47 26 127 30 13 1424.1 25 25.0
247 Leela Chess Zero Gen 14 x64 : 1184.1 200 42 20 138 26 10 1424.1 25 25.0
251 Leela Chess Zero Gen 12 x64 : 1093.4 250 64 26 160 31 10 1276.2 49 40.8
263 Leela Chess Zero Gen 10 x64 : 860.1 92 53 11 28 64 12 654.5 23 23.0
266 Leela Chess Zero Gen 8 x64 : 791.5 92 45 17 30 58 18 654.5 23 23.0
271 Leela Chess Zero Gen 6 x64 : 597.3 92 31 18 43 43 20 654.5 23 23.0
275 Leela Chess Zero Gen 4 x64 : 369.0 150 43 18 89 35 12 622.9 15 15.0
Next I'll wait for network 69 or 70 or whatever it takes to get a new highest elo network, and match it against this new gauntlet and see how it does.
CMCanavessi
on 1 Apr 2018
Id 69 closes the gap to Stockfish Level 10 a lot:
Score of lc_id69 vs sf_lv10: 35 - 50 - 15 [0.425] 100
Elo difference: -52.51 +/- 64.04
Looks like even the 6x64 architecture may still draw level with Stockfish Level 10 after all...
jkiliani
on 1 Apr 2018
I ran a couple round-robin tournaments against Stockfish Level 10, with Id 79 and Id 80, each with and without FPU reduction (and 800 playouts). So far, SF Lv 10 hasn't be surpassed yet, somewhat surprising since the self-play Elo has risen considerably since Id 69:
Rank Name Elo +/- Games Score Draws
1 sf_lv10 61 77 75 58.7% 10.7%
2 lc_id78_fpu -9 70 74 48.6% 24.3%
3 lc_id78 -53 69 73 42.5% 27.4%
Rank Name Elo +/- Games Score Draws
1 sf_lv10 39 51 153 55.6% 17.0%
2 lc_id79_fpu 9 49 153 51.3% 21.6%
3 lc_id79 -48 49 152 43.1% 24.3%
Yesterday I started a new gauntlet with ID 80, which was, in theory, only ~74 elo stronger than ID 69 in self-play, which would translate to 25-30 "real elo" when playing other engines.
After the gauntlet finished, the results are quite a bit better than that:
200 Leela Chess Zero ID 80 x64 : 1821.7 200 76 26 98 45 13 1865.3 25 25.0
213 Leela Chess Zero ID 69 x64 : 1753.8 200 62 21 117 36 11 1865.3 25 25.0
230 Leela Chess Zero Gen 33 (ID 55) x64 : 1669.2 400 185 35 180 51 9 1645.0 45 43.5
252 Leela Chess Zero Gen 30 (ID 44) x64 : 1504.0 200 106 22 72 59 11 1424.7 25 25.0
261 Leela Chess Zero Gen 24 x64 : 1399.7 200 84 22 94 48 11 1424.7 25 25.0
266 Leela Chess Zero Gen 20 x64 : 1316.5 200 64 27 109 39 14 1424.7 25 25.0
279 Leela Chess Zero Gen 16 x64 : 1228.3 200 47 26 127 30 13 1424.7 25 25.0
281 Leela Chess Zero Gen 14 x64 : 1184.4 200 42 20 138 26 10 1424.7 25 25.0
285 Leela Chess Zero Gen 12 x64 : 1093.6 250 64 26 160 31 10 1276.7 49 40.8
297 Leela Chess Zero Gen 10 x64 : 860.0 92 53 11 28 64 12 654.4 23 23.0
300 Leela Chess Zero Gen 8 x64 : 791.4 92 45 17 30 58 18 654.4 23 23.0
305 Leela Chess Zero Gen 6 x64 : 597.2 92 31 18 43 43 20 654.4 23 23.0
309 Leela Chess Zero Gen 4 x64 : 368.9 150 43 18 89 35 12 622.8 15 15.0
Scoring progression:
ID 55: 61.0/200
ID 69: 72.5/200
ID 80: 89.0/200
Even with the flipping bug and the seemingly stalling of the improvement, it's quite a bit better. :)
CMCanavessi
on 3 Apr 2018
Results of round-robin tournaments including Stockfish Level 10, and both the regular and FPU reduction version of Id 80, Id 81 and Id 83 (800 playouts each):
Rank Name Elo +/- Games Score Draws
1 sf_lv10 46 69 84 56.5% 17.9%
2 lc_id80 25 64 83 53.6% 27.7%
3 lc_id80_fpu -72 65 83 39.8% 28.9%
Rank Name Elo +/- Games Score Draws
1 sf_lv10 60 45 200 58.5% 14.0%
2 lc_id81_fpu -5 43 200 49.3% 21.5%
3 lc_id81 -54 45 200 42.3% 14.5%
Rank Name Elo +/- Games Score Draws
1 lc_id83 5 44 200 50.7% 17.5%
2 sf_lv10 3 46 200 50.5% 10.0%
3 lc_id83_fpu -9 43 200 48.8% 19.5%
Stockfish stays ahead until Id 83, where the non FPU reduction Leela Chess draws level with it. FPU reduction has some rather mixed results, being considerably worse for Id 80 but better for Id 81. It seems from results by @jjoshua2 that FPU reduction is actually more effective at higher playouts, but I couldn't confirm that myself.
jkiliani
on 4 Apr 2018
Did you try running 200 or 400 playout tests yet? To see if FPU gets worse at least?
jjoshua2
on 4 Apr 2018
It seems from results by @jjoshua2 that FPU reduction is actually more effective at higher playouts, but I couldn't confirm that myself.
This makes sense, reductions are typically more valuable the deeper the search as time saved increases and risk (of missing something in non-pv) decreases.
jhellis3
on 4 Apr 2018
Id 94 performed surprisingly well against Stockfish Level 10, considering that its self-play Elo is 85 points lower than Id 83:
Rank Name Elo +/- Games Score Draws
1 lc_id94 38 45 200 55.5% 16.0%
2 sf_lv10 -12 46 200 48.3% 11.5%
3 lc_id94_fpu -26 44 200 46.3% 18.5%
It seems that Id 94 may actually be stronger than 83, although a direct match would be required to say for certain. Against, the FPU reduction version did not perform well. As a result, FPU reduction will be temporarily disabled (#254) until a configuration is found that provides a consistent strength benefit.
jkiliani
on 6 Apr 2018
And as soon as FPU reduction is deactivated, I find a network where it gives a benefit again with current parameters:
1 lc_id101_fpu 58 43 200 58.3% 21.5%
2 lc_id101 -12 43 200 48.3% 21.5%
3 sf_lv10 -45 46 200 43.5% 12.0%
Including the performances of both with and without FPU reduction, this is further progress compared to Id 94. It looks like current nets are still getting somewhat stronger.
I will be unable to further test FPU reduction for a while, since I'm going to be testing #267 now, as this seems to have a higher priority for the moment.
jkiliani
on 8 Apr 2018
Gonna post this here:
CMCanavessi
on 8 Apr 2018
Id 103 is a big improvement compared to Id 101 when matched against Stockfish Lv 10. This was a side result of https://github.com/glinscott/leela-chess/pull/267#issuecomment-379577400, where I primarily tested the strength of the decaying temperature implementation.
jkiliani
on 8 Apr 2018
I have started to run a series of 200 game matches on GCP using an 8 processor system.
Very fast time control (all moves in 16 minutes with 0.066 second increment)
Leela uses 8 threads and goes at about 1400-1500 nps
Playing each of the 100 positions in Noomen Short Test Suite as both white and black against Stockfish 8 single CPU Level 9.
I chose this level of Stockfish as it is a reasonable opponent for Leela on this system at this time-control. Essentially at this time control the results will be very sensitive to tactical blunders by Leela.
A match takes about 90 minutes. Here are the results so far. I will update about once a day.
I've given elostat a nominal start rating of 2000 for its calculations, but clearly it is the relative values which matter.
gsobala
on 9 Apr 2018
Can you calculate those ratings with Ordo instead of Elostat? As the latter is severly flawed (still useful to have an idea though).
CMCanavessi
on 9 Apr 2018
As all the games are against the same opponent it probably does not make much difference. Here is Ordo output:
That is about 3.25 ELO per new network ID
gsobala
on 9 Apr 2018
I just did some computationally super cheap tests on the scaling of Id 118, a recent net that I also used for some other tests (that one was for temperature decay, with 800 visits):
Rank Name Elo +/- Games Score Draws
1 lc_id118 80 44 200 61.3% 21.5%
2 lc_id118_d200 53 42 200 57.5% 27.0%
3 sf_lv10 -137 48 200 31.3% 13.5%
So Id 118 is ~200 Elo stronger than Stockfish Level 10.
The tests I did now were on the raw net Id 118, i.e. using only one visit:
Score of lc_1 vs sf_lv0: 154 - 14 - 32 [0.850] 200
Elo difference: 301.33 +/- 57.02
Score of lc_1 vs sf_lv5: 63 - 95 - 42 [0.420] 200
Elo difference: -56.07 +/- 43.38
Judging from this, and the earlier tests I did on Stockfish with skill levels 0, 5 and 10, Leela with just its policy eval is around 800 Elo weaker against Stockfish than it is with 800 visits. This is roughly consistent with the (internal) scaling test I'm currently running on Leela chess, Id 118, and will add later. Easy mode on play.lczero.org is not going to remain all that easy for the average club player, once we move to the 10 block net. At some point soon, the raw net alone should be able to challenge most chess enthusiasts.
Results of the scaling test:
Rank Name Elo +/- Games Score Draws
1 lc_800 557 118 180 96.1% 5.6%
2 lc_400 342 75 180 87.8% 5.6%
3 lc_200 196 57 180 75.6% 7.8%
4 lc_100 105 50 180 64.7% 10.6%
5 lc_50 21 48 180 53.1% 11.7%
6 lc_20 -17 49 180 47.5% 7.2%
7 lc_10 -161 50 180 28.3% 18.9%
8 lc_5 -241 58 180 20.0% 12.2%
9 lc_1 -321 69 180 13.6% 8.3%
10 lc_2 -325 69 180 13.3% 8.9%
No real surprises here, except that the second visit appears to actually cost strength, and that scaling at higher playouts is actually better than at lower.
jkiliani
on 12 Apr 2018
A few more recent tests of raw net against handicapped Stockfish:
Score of Id_125 vs sf_lv5: 89 - 213 - 98 [0.345] 400
Elo difference: -111.37 +/- 30.63
Score of Id_150 vs sf_lv5: 133 - 174 - 93 [0.449] 400
Elo difference: -35.74 +/- 29.97
Id 150 is a slight gain compared to Id 118, while Id 125 is actually a regression, when comparing only the raw networks without playouts. It seems the playing strength improvement is mostly based on better playout scaling of the 10 block nets. Since other tests indicate that Id 149-150 may be similarly strong as Id 125, this also suggests that Id 125 scales very well.
jkiliani
on 20 Apr 2018
Interesting, are those with -p1 or -v1 or something like that?
CMCanavessi
on 20 Apr 2018
I used nodes=1 here in cutechess-cli, which I assume is passed on to lczero as --visits=1.
jkiliani
on 20 Apr 2018
Id 187 is the first net I tested that actually beat SF level 5 on 1 visit:
Score of Id_187 vs sf_lv5: 166 - 138 - 96 [0.535] 400
Elo difference: 24.36 +/- 29.76
Since the reported strength gain at 800 visits since the last 64x6 nets is much more than that, I would conclude also that current nets actually scale better than old ones.
jkiliani
on 25 Apr 2018
I haven't tried this in a while: LCZero at 1 visit against Stockfish Lv 5:
Score of Id_258 vs sf_lv5: 230 - 77 - 93 [0.691] 400
Elo difference: 140.01 +/- 31.51
The pure net is getting considerably stronger, although at a slower rate than Leela with tree search is. Was prompted to make this test by Kingcrusher's experiences on easy mode 😄
jkiliani
on 7 May 2018
Out of curiosity, is anyone still running these tournaments/elo tests? If so, would be nice to see the current state :)
Also, I wonder what is the highest sf level that can be beaten by the pure net and how this changes over time.
As I would like to test some things myself, I was wondering:
How do you run a tournament on cutechess with lczero using a fixed node num and the engines another time control?
it seems that setting a minute per game time control overrides the visits parameter I try to set to lczero, while limiting the number of nodes affects all engines, making them much weaker than their respective levels...
I'm sure it's something to do with the command lines sent... Just the way I'm doing it doesn't seem to work.
grolich
on 17 May 2018
Related issues
mbabigian
·
9Comments
ASilver
·
10Comments
theseapples
·
13Comments
jkiliani
·
10Comments
killerducky
·
18Comments
Most helpful comment
Running a tournament right now with 24 engines under ~1000 elo, Leela Chess Zero Gen 6 is playing. Will update later with how it's doing. Should do somewhat better than Gen 4, but can it beat NEG and Acqua consistently now? Can it at least draw with Easy Peasy? We'll see...