Stockfish: Test scaling at Threads=6 versus Threads=1

[Vinvin20]

+290 Elo for 6 cores over 1 core, very good news !
In the process I got games where 1 core beat 6 cores from a worse position, that will be interesting to improve the eval !

Score of stockfish-windows-1-core vs stockfish-windows-6-cores: 52 - 307 - 15 [0.159]
ELO difference: -289.24 +/- 46.90

374 of 1040 games finished.

[ddugovic]

Wouldn't we expect 6 cores to win?

I got games where 1 core beat 6 cores from a worse position

I hope this is reproducible, even after I merge #153.

[Vinvin20]

Some points to consider about this :

• +290 Elo is very high compare to chess ( In chess, 6 * nodes = around +100 Elo).
• The match will continue for 10 more hours
• The opening lines ( from ccva-520start.pgn in https://sites.google.com/site/zhassociation/download ) are sometimes very unbalanced (position is +2 at first engine move), I hope this test will help to improve this book lines.
• Sometimes 6-cores get >+3 score and lost, probably because of a bad eval or bad search (my point above).
• I saw your King-Safety improvement, that's very good, I'll probably redo the same test with this patch !

[ddugovic]

Sometimes 6-cores get >+3 score and lost, probably because of a bad eval or bad search (my point above).
I saw your King-Safety improvement, that's very good, I'll probably redo the same test with this patch !

This is interesting! Evaluations don't necessarily need to make sense for humans but these games may help us understand failings of the engine.

[Vinvin20]

I forgot to mention the time control is 30sec+0.5sec.

[sf-x]

In chess, 6 * nodes = around +100 Elo

Not at the time control and set of opening positions you used for crazyhouse. At least 150.

[Vinvin20]

Yup, I mismatched numbers (I remember wrongly that "nodes x4 = 60 Elo", but it's "nodes x2 = 60 Elo" )

[ianfab]

I have run a similar test, but with different conditions: 2 vs. 1 core, time control 10+0.1, different book.

ELO: 104.87 +-21.9 (95%) LOS: 100.0%
Total: 1000 W: 623 L: 330 D: 47

[Vinvin20]

+100 Elo for x2 is big but that's very close to my numbers.
The original idea was to test the multi-thread is good for SF-zh. And it's very good !

And I forgot to mention that, on my computer, 1 core speed is around 1.5 Mn/s and 6 cores speed is around 9 Mn/s .

[Vinvin20]

I stopped the match after 780 games :

Score of stockfish-windows-1-core vs stockfish-windows-6-cores: 123 - 630 - 27 [0.175]
ELO difference: -269.37 +/- 31.25

780 of 1040 games finished.

Games are here : https://www.dropbox.com/s/w0j3vvm48viu42n/SF-zh-6cores-vs-1core.zip?dl=0

To find games where "6-cores" blundered, search for games where "1-core" got black and wins.

[MichaelB7]

That's very good. 6x speed should produce about 260 ELO gain - with out even considering the efficiencies using smp. Small sample size , but smp is obviously working very well here.

Sent from my iPhone

On Dec 6, 2016, at 1:59 PM, Vinvin20 [e-mail] wrote:

I stopped the match after 780 games :

Score of stockfish-windows-1-core vs stockfish-windows-6-cores: 123 - 630 - 27 [0.175]
ELO difference: -269.37 +/- 31.25

780 of 1040 games finished.
Games are here : https://www.dropbox.com/s/w0j3vvm48viu42n/SF-zh-6cores-vs-1core.zip?dl=0

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[ianfab]

I have re-run my test (2 cores vs. 1 core, 10''+0.1'') for standard chess and the results are

ELO: 97.69 +-13.9 (95%) LOS: 100.0%
Total: 1000 W: 352 L: 78 D: 570

For reasons of clarity I repeat the crazyhouse results:

ELO: 104.87 +-21.9 (95%) LOS: 100.0%
Total: 1000 W: 623 L: 330 D: 47

[Vinvin20]

Some stats on my test from Ferdinand Mosca here : http://www.talkchess.com/forum/viewtopic.php?p=697971#697971

Ave depth and time comparison:
```
nr player games pts% sumTime aveDep aveTime

1 stockfish-windows-6-cores 781 82.5 00:10:17:33 14.49 00:01:211
2 stockfish-windows-1-core 781 17.5 00:10:39:52 13.41 00:01:266

file : SF-zh-6cores-vs-1core.pgn
score window : [-1000.0, +1000.0]
notes : 1. games and pts% are not affected by score window.
2. Table is sorted by aveDep in descending order.
3. aveTime is the average time/move in m:s:ms
4. sumTime is in d:h:m:s

At winning score say +3 to +6, this is the win probability:

file : SF-zh-6cores-vs-1core.pgn
score window : [+3.0, +6.0]
games and pts% are not affected by score window

nr player games pts(%) winProb
1 stockfish-windows-6-cores 781 82.5 0.96
2 stockfish-windows-1-core 781 17.5 0.62
```

The 6-core is high but could not get 100% even with that score window. But of course the TC is also important to be considered here.

//////////////////////////////////////////////////////////

My comments :

• 1 core to 6 core gives 1 ply deeper.
• for 6 cores when score is [+3.0, +6.0] -> 96% winning score
• for 1 core when score is [+3.0, +6.0] -> 62% winning score

[ddugovic]

Theory-wise, a standard chess game has roughly 36 legal moves in any middlegame position and a branching factor of 6. I suspect that in crazyhouse, the typical number of legal moves per position is closer to 144 (and assume a branching factor nearer to 12).

So when we naively assume:

• 6x speed should produce about 260 ELO gain
• 6x speed does produce depth increase of 1 (assuming the tree has similar shape at Threads=6 as at Threads=1)
• hash table size is not mentioned anywhere in this thread - it would be interesting to gather statistics and know if more threads are used, should the table size increase; and for Threads=1 what is a value that causes a hit rate similar to the hit rate in normal chess

it's possible some of our specific numbers may be inaccurate!

[Vinvin20]

Theory-wise, a standard chess game has roughly 36 legal moves in any middlegame position and a branching factor of 6. I suspect that in crazyhouse, the typical number of legal moves per position is closer to 144 (and assume a branching factor nearer to 12).

Stockfish-chess has a branching factor around 1.5 : http://www.talkchess.com/forum/viewtopic.php?p=694040#694040
Based on difference 6 cores - 1 core, Stockfish-crazyhouse would get a BF around 5.
Numbers would be more accurate with 1 core and time * 6.

hash table size is not mentioned anywhere in this thread

I used 512 MB for 6-cores and same value (512 MB) for 1-core

[ddugovic]

In computing, tree data structures, and game theory, the branching factor is the number of children at each node. That person is referring to EBF:

The effective branching factor (EBF), related to iterative deepening of depth-first search, is conventionally defined as average ratio of nodes (or time used) revisited of the current iteration N versus the previous iteration N-1.

whereas I was referring to "how many candidate moves are there in a given position" irrespective of how much effort is consumed examining those moves.

[ddugovic]

FYI @Vinvin20 I now created a wiki for sharing information.