Stockfish: What conditions should be used for testing changes?

[sf-x]

[ddugovic]

Seeking input from others as testing isn't yet my strong suit...

[ianfab]

So far all my tests use 100ms increment and base times between 100ms and 10s (to avoid repeated games). I will probably switch to always using 10s base time as soon as I have opening books for all variants. I think that using conditions similar to the ones on fishtest is reasonable.

I am open for suggestions on time control, SPRT bounds, etc., but we should always consider the very limited resources compared to fishtest.

[Vinvin20]

We can ask SF-dev-team to use their base http://tests.stockfishchess.org/tests
... half joke

[ddugovic]

@ianfab writes:

In the following I am going to summarize my current thinking on testing and the way I have been doing tests so far. Feedback and suggestions are very welcome.

Steps

Just like on fishtest, I think there should be a first test on STC, and then a second test to show the scaling to LTC, since we have already seen that not doing LTC tests can cause regressions to pass undetected.

Time controls

Using the same time controls as fishtest (STC 10s+0.1s, LTC 60s+0.1s) would probably take too long for LTC, so it could make sense to half/reduce the time controls. Reducing both STC and LTC could restrict the validity of STC tests, whereas only reducing LTC would make it more difficult to see scaling effects, so I am currently not sure about that. Since I did not have opening books for all variants until two days a ago, I have varied the base time (0.1-10s) to avoid repeated games, but used the the same increment as STC on fishtest (0.1s).

Books

Since we are testing many variants, I think we need an automatic way to generate the opening books. Since Stockfish might be the only (strong) engine for some variants, it makes sense to use it for generation. I have started to write such a book generator based on Stockfish two days ago, but of course it is rather experimental yet. Using it, I generated a set of EPD opening books for the variants supported by Stockfish (excluding Relay chess, since I have not dealt with it yet).

Testing at STC

Since I do not know of another way to perform SPRT tests for variant engines, I use my very basic far-from-perfect testing script. Nevertheless, it at least seems to work quite well in pratice so far, since my patches tested with this script improved Stockfish by several hundred Elo in several variants.

So far I have always used [0,20] Elo bounds for SPRT tests. I got to those empirically by considering the typical Elo gains of patches and the number of games neccessary for such patches to pass an SPRT test. In the beginning almost all patches added new code/ideas and the Elo gain typically was huge (>50 Elo), so this was fine, but since more and more patches are going to be paramter tweaks and simplifications with only small Elo differences, and since Elo differences are decreasing as Stockfish is improving, we have to think about the SPRT bounds. Maybe we could simply use a multiple (3-5?) of the bounds used on fishtest (general [0,5], tweaks [0,4], simplifications [-3,1]).

Testing at LTC

Since LTC tests could take a long time, we could think about using a fixed number of games for LTC to only regression test the changes. Probably time will tell whether lengthy tests on LTC are feasible.

Tuning

I have written a slightly modified version of Stockfish's SPSA tuner to add variant support. It should be more or less self-explanatory if you know how to use the SPSA tuner for official Stockfish.

[ddugovic]

Thanks, although I am unfamiliar with SPRT I believe that is excellent guidance. I studied statistics both at university and in high school, I simply am unfamiliar with this particular heuristic/experiment.

Due to official-stockfish/Stockfish#603 and official-stockfish playing in competitions with "Move Overhead=1000", I think LTC (and possibly STC?) tests should be performed with "Move Overhead=1000" and a minimum increment of 1s. It baffles me that fishtest would use different conditions although perhaps official-stockfish/Stockfish is less prone to timeout than this fork?

I have started experimenting with your modified SPSA tuner and submitted a PR addressing most of the confusion I encountered trying to use it.

[sf-x]

My ideas, as an Ideas Guy(TM):
It's essential to keep results per side (if playing without book) or results of game pairs with the same start position (if using a large set of start positions).. Not doing that results in overestimation of statistical error (see http://talkchess.com/forum/viewtopic.php?t=61105&highlight=gsprt ) and waste of testing resources. Why doesn't mainline Stockfish testing doesn't do that? - well, they found enough dupes to contribute their resources.

[ianfab]

@sf-x In principle, this is a good point. However, if we use balanced positions and large opening books, the effect should be rather small, I guess. On fishtest, you additionally have to take into account that results might differ on different machines. Furthermore, I am not sure whether you could absorb (an expectation value of) the differences into the LLR bounds or other parameters and simply reinterpret these parameters. So I do not understand enough of it yet to be able to judge whether this would change much in practice.

[ianfab]

@ddugovic So far there have been only very few time losses in my tests (reported at the end of a test), probably because my testing script takes the times from the output of the engine and does not measure the thinking time itself. This of course assumes the engine to be honest, but as long as a patch does not change the output of thinking time, this should not make a difference.

I have not done much testing with CuteChess or the like, so I can not say much about whether there would be time losses.

[sf-x]

However, if we use balanced positions and large opening books, the effect should be rather small, I guess.

In the linked thread, for _allegedly_ balanced positions in regular chess it was found that error estimated with the common (broken) method was 1.2x bigger than estimate from game pairs. Is that small?

On fishtest, you additionally have to take into account that results might differ on different machines.

How can this be taken into account?

Furthermore, I am not sure whether you could absorb (an expectation value of) the differences into the LLR bounds or other parameters and simply reinterpret these parameters.

???

[ianfab]

However, if we use balanced positions and large opening books, the effect should be rather small, I guess.

In the linked thread, for allegedly balanced positions in regular chess it was found that error estimated with the common (broken) method was 1.2x bigger than estimate from game pairs. Is that small?

Well, 20% is something, but it is not a huge difference and I would be cautious to call a well working method "broken" even if it is not entirely correct from a theoretical point of view.

On fishtest, you additionally have to take into account that results might differ on different machines.

How can this be taken into account?

If you want to take into account correlations, you should consider factors that influence the correlations, right?

Furthermore, I am not sure whether you could absorb (an expectation value of) the differences into the LLR bounds or other parameters and simply reinterpret these parameters.

???

I do not know whether this applies in this case, but if you can get the same or similar results with the current model by simply redefining existing parameters, a new model does not really make a difference. Then only the interpretation of the parameters, but not the calculation itself would be wrong.

[ddugovic]

Currently this is my best LTC test command using cutechess-cli:

cutechess-cli -variant crazyhouse -engine cmd=./stockfish-x86_64-bmi2-kingsafety -engine cmd=./stockfish-x86_64-bmi2-master -each proto=uci tc=240/60+10 "option.Move Overhead=1000" -rounds 200 -concurrency 4 -openings file=books/crazyhouse.epd format=epd order=random -repeat -pgnout test.pgn min

[ianfab]

@ddugovic While long time controls are desirable, I do not think it is practical to use this time control, because it often requires much more than 200 games to get a statistically significant result.

Because of the recently generated opening books, I have switched to using constant base times. I am currently running tests on #145 with different time controls (5+0.05, 10+0.1, 30+0.3, 60+0.6, 120+1.2) and the new opening book to get an idea of which STC and LTC could be used to test for scaling. Of course, one patch is not that much data, but the patch seems to perform well at short time controls while scaling is very bad, so it is a great test case.

Here are the results of tests I started yesterday with the old settings of varying base times:

0.1-10+0.1
LLR: 3.00 (-2.94,2.94) [0.00,20.00]
Total: 740 W: 388 L: 317 D: 35

0.6-60+0.6
LLR: -3.00 (-2.94,2.94) [0.00,20.00]
Total: 298 W: 118 L: 158 D: 22

I will gradually add other results as soon as tests finish.

5+0.05
LLR: 3.01 (-2.94,2.94) [0.00,20.00]
Total: 766 W: 402 L: 330 D: 34

10+0.1
LLR: -2.99 (-2.94,2.94) [0.00,20.00]
Total: 4652 W: 2250 L: 2166 D: 236

30+0.3
LLR: -2.95 (-2.94,2.94) [0.00,20.00]
Total: 210 W: 79 L: 122 D: 9

60+0.6
LLR: -2.95 (-2.94,2.94) [0.00,20.00]
Total: 276 W: 111 L: 152 D: 13

120+1.2
LLR: -3.05 (-2.94,2.94) [0.00,20.00]
Total: 302 W: 123 L: 165 D: 14

[ddugovic]

Thanks and that makes sense, and now I'm using that book. (On my own machine in the future I will test for scaling at 30+1 before 60+1; certainly I could test at 30+0.3 etc. to make tests finish slightly faster but I prefer +1 for my own testing.)

[sf-x]

If you want to take into account correlations, you should consider factors that influence the correlations, right?

I have no idea how to measure them. It seems to me that the effect can be adequately explained by assuming the the game results are not identically distributed. The proposed refinement is to assume (still wrongly, but a lot of data would need to be collected to to correct for it!) that the SUMS of scores in a game pair are identically distributed.

However, if we use balanced positions and large opening books, the effect should be rather small, I guess

I have played 32 game pairs from 32 different positions with @ianfab 's new book (one engine with #153 and other without), and 22 (IIRC) were won by the same side (two White wins or two Black wins). Does anybody else have data for that?

[arbolis]

Wouldn't it be possible to fork fishtest and run it "for free" (only electricity bill)? So that we could let run a few different machines for each patch.

[ianfab]

@arbolis If I remember correctly fishtest uses cutechess, and cutechess does not support many of the variants yet, but this is changing. It would of course be awesome to have such a testing platform.

[ianfab]

@sf-x Sorry, I only skimmed the discussion on talkchess and misunderstood it. I do not have any data for that.

[HGMuller]

Note that WinBoard supports a 'Monte-Carlo book mode', which can be used with a randomizing engine, or a group of equally strong (or time-handicapped to become equally strong) engines. In this mode it decides the frequency with which moves should be played based on the statistics of their prior results, selects the book move that is most under-played, and lets the engine think (in the hope it will generate a new move) when all moves are played (approximately) with the frequency they deserve. (Note that the latter will always be the case if there is only one move for the position.) Just play a tournament with the additional option -mcBookMode true. You can later convert the PGN file of the tourney to a book.
Instead of a randomizing engine, you can also do a number of self-play matches at various time controls. Because WinBoard would be feeding moves that were generated by the engine at one TC to engines playing at another TC it even makes sense to do the same TC several times (with other TCs in between).

[sf-x]

I'll leave this for future reference: http://hardy.uhasselt.be/Toga/GSPRT_approximation.pdf

[arbolis]

Should we do some regression tests from time to time? Just to make sure we're heading toward the right direction (id est elo increase).

[ianfab]

Sure, it makes sense to do regression tests. The main questions are how often and with which time control. Regression tests should probably be based on release versions.

[arbolis]

I see. Maybe after X functional patches? Where X could be 10 or so.

[ianfab]

I think this is difficult to count, since there are patches for several variants and also upstream changes. I will probably simply use the release versions something like every two or four weeks and go for 1000 games per variant on 10+0.1, where it should take about one or two days to do regression tests for all variants on one machine. Regression tests on longer time control take much longer, so I will do them less often. If I have time to write an automatization script soon, first results should be there in a few days.

Update: I am now running regression tests for fishnet-091216 vs. fishnet-301116.

[ddugovic]

Periodically running the SPSA Tuner is probably a good idea as well. I am tuning razor_margin at the moment...

[ianfab]

Yes, e.g. I have re-tuned piece values several times (especially in the case of antichess and crazyhouse, where two more tuning sessions are currently running) when there had been significant changes to the evaluation function. It of course also makes sense for search parameters, especially since the razoring margin has not been tuned yet (or to be precise only hand-tuned) for crazyhouse.

[ianfab]

@arbolis First regression test results are there now.

[Vinvin20]

I'll run a big regression test (161130 vs 161209) in Crazyhouse on my 6 cores@4GHz :
55s+2s
50s+2s
45s+2s
40s+2s
35s+2s
30s+2s

I like big increments because ZH is less linear (eval from move to next move) than chess, so each move needs a new deeper consideration.

[ddugovic]

To be clear @Vinvin20 is referring to my comment on #131 where I referenced #161 and asked him to test on the Windows platform.

[ddugovic]

It just now occurred to me that SPSA Tuner-based tests (of Stockfish) can be reposited! I intend to add one test per variant.

[ianfab]

@ddugovic What do you mean by that?

[ddugovic]

Sorry I was unclear. So far I have added crazyhouse.conf and crazyhouse.var. It now occurs to me that having one .conf and one .var file per variant (which tune all the parameters) seems like the most efficient way to use the tuner.

[ddugovic]

Last year I attempted a fishtest install. I will attempt it again.

[ianfab]

@ddugovic This would be great. With small changes this could at least be used for the variants supported by CuteChess and probably for all variants regarding tuning.

[Vinvin20]

I'll run a big regression test (161130 vs 161209) in Crazyhouse on my 6 cores@4GHz :

Results are in "Regression test results" : https://github.com/ddugovic/Stockfish/issues/170#issuecomment-266518756

[ianfab]

Last year I attempted a fishtest install. I will attempt it again.

@ddugovic Here is my partly successful attempt on a fishtest server and worker: http://35.161.250.236:6543/tests. It is only a first attempt, so many things do not work and/or have not been tested yet, but at least some basic functionalities are supported for variants (an intentionally bad patch for atomic chess indeed shows bad results, so it seems to be working^^). I have not added opening books for variants yet, so only variants with the same FEN format as standard chess are supported, except for giveaway, which is not supported by cutechess.

The currently running worker is an AWS EC2 c3.large spot instance (with Ubuntu 16.04), initialized with a script based on https://github.com/glinscott/fishtest/wiki/Running-the-worker-in-the-Amazon-AWS-EC2-cloud, but adjusted to be able to use a cheap instance for testing purposes. In case you want to try to add another instance, you can find the script below. You only have to replace USERNAME and PASSWORD by an account you created on http://35.161.250.236:6543/signup (and optionally adjust the number of cores in the for loop).

#!/bin/bash
# replace USERNAME and PASSWORD with your fishtest username and password
# the double quotes deal with symbols in username or password: don't delete them
username="USERNAME"
password="PASSWORD"

# update software
apt update -y
#apt full-upgrade -y
apt install -y python build-essential libqtcore4 libqt5core5a unzip
useradd -m fishtest
sudo -i -u fishtest wget https://github.com/ianfab/fishtest/archive/master.zip

# disable hyperthreads
for cpunum in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | cut -s -d, -f2- | tr ',' '\n' | sort -un); do
     echo 0 > /sys/devices/system/cpu/cpu$cpunum/online
done

# generate a script for running fishtest
cat << EOF > runscript.sh
#!/bin/bash
partitionid=0
# partition the available cores in independent fishtest workers
# the sum of the partitions should be the number of physical cores -1
for partition in 1
do
  partitionid=\$((partitionid+1))
  mkdir part_\$partitionid
  cd part_\$partitionid
  unzip ../master.zip
  echo "make build ARCH=x86-64 COMP=gcc -j \$partition" > fishtest-master/worker/custom_make.txt
  (
  # put fishtest in a loop restarting every 4h to deal with eventual hangs or crashes
  while [ true ]; do
    # some random delay to avoid starting too many fishtest workers at the same time
    sleep "\$((RANDOM%20)).\$((RANDOM%100))"
    timeout 4h python fishtest-master/worker/worker.py --concurrency \$partition "$username" "$password" >& out
  done
  ) &
  cd ..
done
wait
EOF
chmod a+rx runscript.sh

# execute script
sudo -i -u fishtest /bin/bash `pwd`/runscript.sh

# if we come here, terminate
#poweroff

[ianfab]

So far most things are working on my fishtest instance http://35.161.250.236:6543/tests, so I invite you to create tests and/or add workers. Do not hesitate to ask in case of questions, especially if you want to contribute a worker.

What is not working yet:

• giveaway is not supported by cutechess yet, so it can not be tested
• the UCI_Variant option values of Stockfish and cutechess are incompatible in the case of three-check chess (threecheck/3check), so it can only be tested using a workaround
• I compiled the latest cutechess (with support for UCI_Variant option and many variants) only for linux 64bit, so I would need the respective executables of cutechess before you can add a windows or linux 32bit worker

What is working:

• testing and SPSA tuning for all (main) variants except giveaway and three-check
• adding linux 64bit workers (e.g. AWS EC2 instances)

[Vinvin20]

Great achievement, @ianfab ! Thank you !

[lantonov]

Congratulations for setting up fishtesting!
I tried to add my computer as worker, changed server ip and port in fishtest (config files and worker.py), created username and password in your network, but was not successful because the updater comes into a loop trying to change from version 59 to 61.
My OS is Windows 10.

[ianfab]

@lantonov Thanks for your support. So far I have only added AWS EC2 linux machines as workers, but I will try to help you to get it working on windows.

The first important point is that I had to update the cutechess compiles, because the old version used for fishtest does not support the UCI_Variant option (and many of the variants). So far I have only updated the linux 64bit compile, so it would be nice if you could compile the latest cutechess-cli on windows and open a pull request with an updated cutechess-cli-win.zip in https://github.com/ianfab/FishCooking.

Furthermore, you have to use the worker version from https://github.com/ianfab/fishtest, because I had to apply some changes to get it working for variants and I adjusted URLs (FishCooking repo, etc.), IP and port.

With these two changes, I think it should work. Please only add a windows machine as soon as the cutechess compile in https://github.com/ianfab/FishCooking is updated, because the worker will otherwise only play standard chess regardless of which variant is stated in the test (since the UCI_Variant option command is not send).

[theo77186]

@ianfab Done! See https://github.com/ianfab/FishCooking/pull/1
EDIT: it is 32-bit so it will work for all Windows versions.

[lantonov]

Thanks, Theo! I tried but cutechess gave me troubles compiling.

[lantonov]

In Fishtest now, playing horde at the moment. However, I entered with 3 cores on one computer and although it registered them as 3 computers of 1 core, only 1 core is playing and the other 2 cores stand idle.
P.S. Got it right at last. The problem was in the location of my fishtest.cfg

[lantonov]

For losers, cutechess gives illegal moves. Variants crazyhouse and horde play normally, though there are 5/100 games lost on time in zh.
Although there is warning for illegal moves in losers, it plays the games to the end, and the pgn looks normal.

[ddugovic]

If possible, maybe try configuring Move Overhead=1000 like official-stockfish does for competitions.

[ianfab]

Thank you @theo77186 for providing the windows binary, I have just merged your PR.

And thank you @lantonov for the feedback. Let's wait for more games to be finished to get more statistics for the time losses. If the rate remains that high, move overhead might have to be increased as suggested by @ddugovic, even though I think increasing it from 30 to something like 100 should be enough to reduce the rate of time losses to a manageable level.

[lantonov]

"No tasks available at this time, waiting..." although there are pending tasks. Also, the previous worker sessions are not closed. These may be connected.

[theo77186]

@lantonov: as there are 4 reported (not terminated correctly) connections, the limit strikes and it prevents further connections.
EDIT: unfortunately, these ghost workers cannot be killed. There is a bug in the fishtest implementation as workers are normally killed after about 1-2 hours.

[ianfab]

For now I increased the limit of workers to 8. I will look into it later to see how to fix this.

[ianfab]

I found out that there is a script to remove inactive workers (on the server side), so the problem should be solved for now.

[ddugovic]

On Windows 10 I installed the Linux Subsystem for Windows and under this Linux I observe:

Step 4/4. Deleting profile data ...
make ARCH=x86-64-modern COMP=gcc profileclean
make[1]: Entering directory `/tmp/tmp5E0t2D/ddugovic-Stockfish-1d5e15a/src'
make[1]: Leaving directory `/tmp/tmp5E0t2D/ddugovic-Stockfish-1d5e15a/src'
Verifying signature of stockfish ...
Verifying signature of base ...
CPU factor : 0.808452 - tc adjusted to 8.08+0.08
Running tune_atomic_close_enemies vs tune_atomic_close_enemies
['/mnt/c/Users/Gaming/Desktop/fishtest/worker/testing/cutechess-cli', '-repeat', '-rounds', '2', '-tournament', 'gauntlet', '-srand', '1144930363', '-resign', 'movecount=8', 'score=800', '-draw', 'movenumber=34', 'movecount=8', 'score=20', '-concurrency', '1', '-openings', u'file=atomic.epd', u'format=epd', 'order=random', 'plies=16', '-variant', u'atomic', '-engine', 'name=stockfish', 'cmd=stockfish', u'option.Hash=4', u'option.mCloseEnemies[ATOMIC_VARIANT]=15', '-engine', 'name=base', 'cmd=base', u'option.Hash=4', u'option.mCloseEnemies[ATOMIC_VARIANT]=19', '-each', 'proto=uci', 'tc=8.08+0.08', 'option.Threads=1']
/mnt/c/Users/Gaming/Desktop/fishtest/worker/testing/cutechess-cli: error while loading shared libraries: libQt5Core.so.5: ('TC limit', 80.84523695233669, 'End time:', datetime.datetime(2017, 1, 31, 20, 39, 47, 413730))
cannot open shared object file: No such file or directory
['/mnt/c/Users/Gaming/Desktop/fishtest/worker/testing/cutechess-cli', '-repeat', '-rounds', '2', '-tournament', 'gauntlet', '-srand', '4192311519', '-resign', 'movecount=8', 'score=800', '-draw', 'movenumber=34', 'movecount=8', 'score=20', '-concurrency', '1', '-openings', u'file=atomic.epd', u'format=epd', 'order=random', 'plies=16', '-variant', u'atomic', '-engine', 'name=stockfish', 'cmd=stockfish', u'option.Hash=4', u'option.mCloseEnemies[ATOMIC_VARIANT]=15', '-engine', 'name=base', 'cmd=base', u'option.Hash=4', u'option.mCloseEnemies[ATOMIC_VARIANT]=19', '-each', 'proto=uci', 'tc=8.08+0.08', 'option.Threads=1']
/mnt/c/Users/Gaming/Desktop/fishtest/worker/testing/cutechess-cli: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory('TC limit', 80.84523695233669, 'End time:', datetime.datetime(2017, 1, 31, 20, 39, 48, 650214))

[ianfab]

@ddugovic I think there are several ways to fix this. If LD_LIBRARY_PATH is set to the directory where libQt5Core.so.5 can be found (delivered with cutechess-cli-linux-64.zip), either manually by you or inside the fishtest worker code, it should work. Alternatively, installing libqt5core5a should solve it, too. I am however not sure whether there are side effects if I set LD_LIBRARY_PATH inside the worker code, at least it would probably have to check first whether it is under linux.

[theo77186]

@ddugovic @ianfab It seems that the Linux subsystem's linux-ld.so isn't working proprely. Changing LD_LIBRARY_PATH won't get any better at all. Installing libqt5core5a doesn't work either as the cutechess executable seems to be binary-incompatible with the package.
PS: does anyone have issues with my executables for cutechess?
Edit: formating
Edit 2: I have verified on Linux for illegal move issue, it still occurs. It's either SF sending bad PVs or cutechess that have wrong illegal move detection.

[lantonov]

For Losers, I get a warning on each move that it is illegal but otherwise it plays as it should. I think this is a problem in the code, not in the executable.

[ianfab]

The losers implementation is rather new and has not been heavily tested, since it is not used on lichess, but it seemed to work fine, at least I have not observed an illegal move yet. So I suspect that it is an issue with cutechess, but I am not entirely sure. If you observe an illegal PV move, please report it.

[ianfab]

There was a hard-coded value in fishtest I was not aware of when changing the chunk size of tasks, which caused an incorrect calculation of the internal priority. This is why new tests had a much higher internal priority than older tests, which caused old tests to remain in the queue for a long time. It should be fixed now, but feedback on the distribution/priority of tasks is still welcome.

[lantonov]

Chunk size increased from 100 to 200. Is it intentional?

[ianfab]

Yes, it is. Due to the small chunk size, machines were switching frequently between tests, so I thought it would make sense to increase chunk size to reduce the time spent on compiling.

[lantonov]

Right, compilation takes a big part of machine time. But you can greatly reduce the compilation time if you use "build" instead of "profile-build". In my opinion, supported by some observations on performance in Fishtest, using profile-build in testing has no particular advantage over plain build. It has the disadvantage that increases variation between machines, as each machine uses a slightly different compile adjusted to its parameters. There is another, bigger disadvantage of profile-build for some machines (e.g., mine which is Windows 8). The OS (more properly the anti-virus program F-secure) has a defence that I can't remove (to change its settings it requires admin password which I don't have), which requires a user confirmation before starting each new program loaded on the HD. This means that before starting each new compile, a popup box appears that requires me to check "Run the new program on my computer" and then click OK. On simple build this popup does not appear. That's why I can't leave my computer overnight or for longer periods of time with profile-build.

[ianfab]

I agree, and I have also been using build instead of profile-build.

[alwey]

The illegal move topic might well be a cutechess problem. It would be nice to get a few samples of this issue especially for Losers.

For Giveaway and Suicide I observed termination of games with "illegal move" adjudication in 25%-30% of my test games (15 s +150 ms/move). I then created a pull request [WIP] for cutechess with a work-around for Giveaway and Suicide. For these variants the PV will not be converted to SAN and tested for legality any more. The illegal move rate then dropped to 6 out of 2000 and any of these 6 cases still was a false positive. So Stockfish was fine.

[ianfab]

I think that we should use wider bounds than the currently used [-6,2] for testing simplifications, since simplifications are more important compared to official stockfish and small regressions are less problematic. I suggest to use [-10, 5]. What do you think?

[ddugovic]

I like the motivation but I'm confused, isn't SPRT based on equality null hypotheses such that using [-10, 5] would take longer than using [-6, 2]?

[ianfab]

I am not an expert on this, but LLR should be zero for a -2 Elo performance in the case of [-6, 2] and for -2.5 Elo in the case of [-10, 5], so small regressions are more likely to pass the latter and since the hypotheses are more different, the LLR should increase more quickly if the performance is above this threshold.

In order to have some numbers to support this reasoning, I have run this SPRT simulator for both choices of bounds:

[-6, 2]

BayesElo    Elo %Pass   Avg
-10.00  -6.20   0.0027  5540
-9.00   -5.58   0.0060  6280
-8.00   -4.96   0.0118  7237
-7.00   -4.34   0.0248  8460
-6.00   -3.72   0.0498  10010
-5.00   -3.10   0.0977  11962
-4.00   -2.48   0.1867  14003
-3.00   -1.86   0.3239  15734
-2.00   -1.24   0.4992  16481
-1.00   -0.62   0.6770  15813
0.00    -0.00   0.8143  14013
1.00    0.62    0.9010  11913
2.00    1.24    0.9498  10060

[-10, 5]

BayesElo    Elo %Pass   Avg
-10.00  -6.20   0.0473  2892
-9.00   -5.58   0.0701  3163
-8.00   -4.96   0.1033  3460
-7.00   -4.34   0.1454  3789
-6.00   -3.72   0.2023  4090
-5.00   -3.10   0.2720  4371
-4.00   -2.48   0.3562  4635
-3.00   -1.86   0.4524  4692
-2.00   -1.24   0.5503  4740
-1.00   -0.62   0.6463  4642
0.00    -0.00   0.7291  4385
1.00    0.62    0.7998  4080
2.00    1.24    0.8569  3779

As you can see the average number of games is much lower for the latter, but the probability of a regression to pass increases.

[ianfab]

@ddugovic If you agree on the new bounds I would consider this STC test as passed (LLR >> 3 with the new bounds) and suggest you to stop STC and submit LTC with the new bounds.

[ddugovic]

OK, the bounds [-10, 5] make sense to me (small regressions are permissible) and I'll stop that STC test.

[ianfab]

You can now filter tests by variant if you click on a variant name, e.g. http://35.161.250.236:6543/tests/variant/crazyhouse, and the variant is also displayed in the list of test results. I hope this improves clarity.

[ddugovic]

Excellent! One minor note -- if it's convenient to implement, could the page title (Stockfish Testing Queue) indicate the variant filter?

[ianfab]

@ddugovic Good idea. It now displays the value of variant, username and/or success_only below the tittle if a filter is applied.

[ianfab]

I have updated the regression test result page. It now labels the versions by date, adds up relative elo differences of test results, and propagates the errors. I think the graph and table are more clear now.

[ppigazzini]

@ddugovic I added a static cutechess-cli.exe for Windows and a cutechess-cli for Linux built with Qt5 static. Now fishtest runs fine on several platforms: Windows MinGW, Ubuntu 14.04 and newer, CentOS 7, Linux Subsystem for Windows. On Windows 10 LSW is the faster way.

[ianfab]

The queue is running empty more and more often in the last days, because I am busy with other things. It would be great to have more people contributing tests and tuning attempts.

[ppigazzini]

@ianfab @ddugovic : chapeaux, you two guys have done an extraordinary work. In order to try to involve other developers, perhaps some marketing post on lichess, talkchess? IMO it will be useful to have a space where ask questions about the project.

[ddugovic]

In general the Lichess Feedback forum has served as a useful space... though it sounds like people aren't satisfied with that & people want a chat room?

Perhaps for that purpose the Lichess discord will suffice until people who aren't busy create and moderate a dedicated place for discussion...

[Vinvin20]

I already posted some advertising on SF forum :
https://groups.google.com/forum/?fromgroups=#!topic/fishcooking/lvp7FNMRWII

[ddugovic]

Right, although I assume advertisements attract users moreso than developers, and they encourage tens of thousands of active Lichess users reporting issues to bypass the Lichess Feedback forum. That forum is helpful since I don't have an iPad, iPhone, Macbook, etc. and I can't keep up with all the hardware and browser issues like Lichess can.

The other problem is that for new developers there is a learning curve and skilled people tend to be busy; it's not clear what a new developer would do... maybe parameter tuning & testing? But people tend to avoid such tedious work.

[ppigazzini]

I started writing some pages in the Wiki to launching the worker on different OS.

[ianfab]

@ppigazzini Thanks for the kind words and your efforts in extending the wiki.

@ddugovic There is a learning curve, and developing a strong engine can be tedious, but at least in my experience it is possible to get familiar with small parts of the Stockfish code without having to understand much of the rest. When I contributed my first evaluation patches to this repository, I had basically only looked at evaluate.cpp and not much else and I did not have (much) prior knowledge of the Stockfish code. Furthermore, I think developing official stockfish is much more tedious, since it is incredibly difficult to find improvements there, whereas the history of my fishtest instance shows that ~40% (without regression and LTC tests it probably is less, but anyway) of our tests are succesful.

[ppigazzini]

@ianfab I wrote some wiki pages of Official Stockfish so it's a simply cut&paste work with some text reviewing. I will open a couple of PR for mv fishtest in order to use MSYS2 python, simplifying the worker installation on Windows.

[ppigazzini]

@ddugovic @ianfab please review the wiki page Creating my first test. I updated the SPRT bounds checking the tests history, but I don't know all the changes done respect to the official Fistest.
Should be useful to have some background/reference info in Home; if I'm not wrong, Multi Variant Stockfish is used in these lichess.org projects (by @niklasf):

• lichess.org custom Stockfish
• fishnet: distributed Stockfish analysis for lichess.org
• PNaCl Stockfish

[ddugovic]

Thanks, looks great! I have updated the Contributor Guidelines section.

[ppigazzini]

Added Installing a development multi variant fishtest server on Ubuntu with the updated installation script.

[ddugovic]

In terms of strength I am most concerned about popular variants:

Blitz: 80,496 players
Crazyhouse: 3,446 players
Antichess: 2,526 players(!)
Atomic: 1,757 players
Others: < 1,200 players

In light of this and of issue #278 , perhaps we should consider that for future PRs/tests -- other than standard, crazyhouse, antichess, and atomic -- should also not cause more than a 1% slowdown for make build benchmarks. (No current PR is in danger of this, however by this standard #281 seems infeasible.)

[ianfab]

I have slightly adjusted the test submission page, so that the bounds and the select boxes are not reset when changing time control or when rescheduling tests, since I found this behavior a bit annoying. Rescheduling for LTC should now be doable with two or three clicks. Please let me know if it does not work properly.

[ianfab]

I consider updating the cutechess compiles for fishtest with the PR https://github.com/cutechess/cutechess/pull/200 in order to be able to test giveaway and suicide chess on fishtest.

@ppigazzini If it is not too much trouble, could you please compile cutechess with the PR included and open a PR in my fishtest repo?

[ppigazzini]

@ianfab PR opened on your FishCooking repo and your fishtest repo: close the wrong one.

I use linux containers to build and test on different distro/versions of linux.

[ianfab]

@ppigazzini Thanks a lot. I'll merge it, adjust fishtest for the new variants, and submit some tests to verify that fishtest is working for losing chess variants. I think I might have to bump the worker version to force the workers to download the new cutechess compiles.

[ianfab]

Enforcing the update of cutechess compiles on all workers without manually accessing them required multiple updates of the worker version. Sorry if this caused any issues. It should work fine now, and similar changes should cause less problems in the future because of adjustments of the worker code.

I am doing some verification tests for giveaway. The results are looking good so far.

[ianfab]

For ease of use I have added a few buttons to fishtest to set SPRT bounds and the opening book name.

[ddugovic]

FYI, although I am no longer handling merges from upstream, until someone merges upstream changes future PRs will remain open (unmerged).

[ianfab]

FYI, although I am no longer handling merges from upstream, until someone merges upstream changes future PRs will remain open (unmerged).

I can try to do some upstream merges even though I do not like that it sounds like blackmailing, where you could have simply asked me. I can not promise that I will have time to do all merges in addition to filling the queue with tests and maintaining fishtest.

If you are busy, I understand that you do not have enough time to do all the upstream merges, but I am surprised that you announce not to do any upstream merges any more. May I ask what lead you to this decision?

[ddugovic]

I am busy, but also the repository keeps growing as gaining Elo is the only consideration and it is difficult to control for complexity, let alone for quality (in terms of blatantly incorrect perft and search results). I'm not saying anyone has to help, just that the project is growing and as I become busier I don't know what to do with it.

[ianfab]

I am busy, but also the repository keeps growing as gaining Elo is the only consideration

I would claim that my considerations are much more nuanced than that. Just because I evaluate the complexity of the code in a different way than you (not claiming that mine is the "right" way, this simply is subjective), that does not mean that I do not consider it. If this is about losers chess, I can just repeat myself once more that I was mainly concerned about the workflow. Testing guidelines have to be a bit unflexible, because they are mainly there to control the unavoidable bias of developers. Of course one can make exceptions, but only after very careful reasoning in order to retain control of the bias.

... quality (in terms of blatantly incorrect perft and search results).

It has a lot of weaknesses and even some bugs, but I am wondering what you mean by "blatantly incorrect perft and search results".

[ianfab]

Fishtest stopped working a few hours ago due to a crash of mongodb. It should be up again now.

[ianfab]

Sorry, there still seem to be some issues and I will not be able to access the server during the next few hours, but I will try to fix it as soon as possible.

[ianfab]

Unfortunately, neither automatic repairing nor manually deleting corrupted data nor merging a backup helped, so I finally had to completely restore the database from a backup. This means that some data was lost, but at least it seems to be working again. Sorry for the inconvenience.

[ianfab]

Here is a backup of the HTML page, where you can see the test results that have been lost in the database:

Stockfish Testing Framework.html.txt

[ianfab]

When I tuned PSQTs for several variants I wanted to avoid the tedious and error prone work of manually copying hundreds of tuning results into the code, so as a result of this I created a script that automatically replaces the values in the code given the tuning results, which might also be useful for others: https://github.com/ianfab/fishutils. It still has several limitations since it does not actually parse C++ code but only searches for regular expression patterns, but so far it has worked well for me in most cases.

[ddugovic]

Ah, interesting... that means it isn't necessary to rebase from the tune_variant branch (which I periodically rebase upon master).

[ianfab]

@ddugovic I am not sure I understand your comment. What the script does is that it gets an input like param: mLever[4], best: 27.00, start: 17.00, min: -100.00, max: 200.00, c 41.714131, a 223.069148 (copied from tuning results in fishtest), then finds the array Lever in pawns.cpp, and changes the middlegame value of the array entry with index 4 in the code from 17 to 27, so you do not have to do this code change manually (which can be tedious if it is not only one parameter as in the example but maybe hundreds).

[ddugovic]

Oh! Then I entirely misunderstood (I thought we were referring to the tuning instrumentation code - I overlooked "tuning results"). Now that I think more about it, that code is in tune.h and tune.cpp respectively...

Thanks @ianfab and I very much look forward to using your tool!

[ianfab]

FYI: In about 24 hours my fishtest instance will be down for a few hours.

[ianfab]

@ddugovic Recently, your machine keeps picking up tests, but then often does not start to play any games. Could you have a look what is happening there? It might be an issue with the lower bound on nps.

[ddugovic]

EDIT: I've now upgraded from Linux 4.13.0-16-lowlatency to Linux 4.13.0-16-generic. I don't know how my system chose to migrate the kernel (to lowlatency) or whether it makes a difference.

@ianfab I'm not seeing any errors. I just upgraded my machine to Ubuntu 17.10 and just now rebased scale_concurrency... here's a log fragment prior to rebasing (I didn't see anything strange):

Oct 26 00:46:24 dugovic-host python[1010]: ('TC limit', 12121.796171391286, 'End time:', datetime.datetime(2017, 10, 26, 1, 38, 21, 600973))
Oct 26 00:46:24 dugovic-host python[1010]: Verifying signature of stockfish ...
Oct 26 00:46:24 dugovic-host python[1010]: Verifying signature of base ...
Oct 26 00:46:24 dugovic-host python[1010]: CPU factor : 2.006018 - tc adjusted to 20.06+0.20
Oct 26 00:46:24 dugovic-host python[1010]: Running race_movepick1 vs master
Oct 26 00:46:24 dugovic-host python[1010]: ['/home/fishtest/fishtest/worker/testing/cutechess-cli', '-repeat', '-rounds', '500', '-tournament', 'gauntlet', '-pgnout', 'results.pgn', '-srand', '2474883693', '-resign', 'movecount=8', 'score=800', '-draw', 'movenumber=34', 'movecount=8', 'score=20', '-co
ncurrency', '3', '-openings', u'file=racingkings.epd', u'format=epd', 'order=random', 'plies=16', '-variant', u'racingkings', '-engine', 'name=stockfis
h', 'cmd=stockfish', u'option.Hash=4', u'option.Move Overhead=100', '-engine', 'name=base', 'cmd=base', u'option.Hash=4', u'option.Move Overhead=100', 
'-each', 'proto=uci', 'tc=20.06+0.20', 'option.Threads=1']
Oct 26 00:46:24 dugovic-host python[1010]: ('TC limit', 16716.81711802073, 'End time:', datetime.datetime(2017, 10, 26, 3, 57, 46, 128231))
Oct 26 00:46:24 dugovic-host python[1010]: Verifying signature of stockfish ...
Oct 26 00:46:24 dugovic-host python[1010]: Verifying signature of base ...
Oct 26 00:46:24 dugovic-host python[1010]: CPU factor : 1.263886 - tc adjusted to 12.64+0.13
Oct 26 00:46:24 dugovic-host python[1010]: Running race_movepick1 vs master
Oct 26 01:43:04 dugovic-host python[1010]: ['/home/fishtest/fishtest/worker/testing/cutechess-cli', '-repeat', '-rounds', '500', '-tournament', 'gauntlet', '-pgnout', 'results.pgn', '-srand', '2192238218', '-resign', 'movecount=8', 'score=800', '-draTask exited
Oct 26 01:43:06 dugovic-host python[1010]: Config:
Oct 26 01:43:06 dugovic-host python[1010]: debug: 'no'
Oct 26 01:43:06 dugovic-host python[1010]: sanitize: 'no'
Oct 26 01:43:06 dugovic-host python[1010]: optimize: 'yes'
Oct 26 01:43:06 dugovic-host python[1010]: arch: 'x86_64'
Oct 26 01:43:06 dugovic-host python[1010]: bits: '64'
Oct 26 01:43:06 dugovic-host python[1010]: kernel: 'Linux'
Oct 26 01:43:06 dugovic-host python[1010]: os: 'GNU/Linux'
Oct 26 01:43:06 dugovic-host python[1010]: prefetch: 'yes'
Oct 26 01:43:06 dugovic-host python[1010]: popcnt: 'yes'
Oct 26 01:43:06 dugovic-host python[1010]: sse: 'yes'
Oct 26 01:43:06 dugovic-host python[1010]: pext: 'no'
Oct 26 01:43:06 dugovic-host python[1010]: Flags:
Oct 26 01:43:06 dugovic-host python[1010]: CXX: g++
Oct 26 01:43:06 dugovic-host python[1010]: CXXFLAGS: -Wall -Wcast-qual -std=c++11  -pedantic -Wextra -Wshadow -m64 -DANTI -DATOMIC -DBUGHOUSE -DCRAZYHOUSE -DHORDE -DKOTH -DLOOP -DLOSERS -DRACE -DSKILL -DSUICIDE -DTHREECHECK -DUSELONGESTPV -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -flto
Oct 26 01:43:06 dugovic-host python[1010]: LDFLAGS:  -m64 -Wl,--no-as-needed -lpthread -Wall -Wcast-qual -std=c++11  -pedantic -Wextra -Wshadow -m64 -D
ANTI -DATOMIC -DBUGHOUSE -DCRAZYHOUSE -DHORDE -DKOTH -DLOOP -DLOSERS -DRACE -DSKILL -DSUICIDE -DTHREECHECK -DUSELONGESTPV -DNDEBUG -O3 -DIS_64BIT -msse
 -msse3 -mpopcnt -DUSE_POPCNT -flto
Oct 26 01:43:06 dugovic-host python[1010]: Testing config sanity. If this fails, try 'make help' ...

[ppigazzini]

@ddugovic I have the impression that my 15 core worker lost a 10% nps after the upgrade to Ubuntu 17.10 and gcc 7.2.0. In official fishtest sometimes it doesn't start because the nps is under the threshold, never happened in the previous months with Ubuntu 17.04 and gcc 7.1.x. In multivariant fishtest I lowered the threshold according this PR https://github.com/ianfab/fishtest/pull/12

[ddugovic]

I've now switched from my fishtest branch to master. I see in the log file:

Oct 26 08:02:01 dugovic-host python[14167]: Total time (ms) : 5932
Oct 26 08:02:01 dugovic-host python[14167]: Nodes searched  : 5234652
Oct 26 08:02:01 dugovic-host python[14167]: Nodes/second    : 882443
Oct 26 08:02:01 dugovic-host python[14167]: Step 3/4. Building optimized executable ...

and presumably the optimized Stockfish is even faster. I'm building with latest gcc-7 and with the BMI2 architecture flag.

[ppigazzini]

@ddugovic if I'm not wrong your worker doesn't work yet. Start the worker from the cli python worker.py --concurrency 2 ddugovic "<password>" to try to view where the script stops.

[ddugovic]

@ppigazzini OK, I'm running it that way... as far as I can tell, everything runs normally, 2 cores are running Stockfish, no errors are logged. I've started to look at source code trying to figure out where to add verbose logging code.

Step 4/4. Deleting profile data ...
make ARCH=x86-64-bmi2 COMP=gcc profileclean
make[1]: Entering directory '/tmp/tmp_jb_Ns/ianfab-Stockfish-6cc33bb/src'
make[1]: Leaving directory '/tmp/tmp_jb_Ns/ianfab-Stockfish-6cc33bb/src'
Verifying signature of stockfish ...
Verifying signature of base ...
CPU factor : 1.236377 - tc adjusted to 37.09+0.37
Running 3check_tempo vs master
['/home/fishtest/fishtest/worker/testing/cutechess-cli', '-repeat', '-rounds', '500', '-tournament', 'gauntlet', '-pgnout', 'results.pgn', '-srand', '2465764753', '-resign', 'movecount=8', 'score=800', '-draw', 'movenumber=34', 'movecount=8', 'score=20', '-concurrency', '2', '-openings', u'file=3check.epd', u'format=epd', 'order=random', 'plies=16', '-variant', u'3check', '-engine', 'name=stockfish', 'cmd=stockfish', u'option.Hash=32', u'option.Move Overhead=100', '-engine', 'name=base', 'cmd=base', u'option.Hash=32', u'option.Move Overhead=100', '-each', 'proto=uci', 'tc=37.09+0.37', 'option.Threads=1']
('TC limit', 46364.125294412195, 'End time:', datetime.datetime(2017, 10, 27, 17, 1, 33, 529399))

[ianfab]

@ddugovic The last line of the log is printed at https://github.com/ianfab/fishtest/blob/9e76acbbac1af1e4e22953f22efc91fbb571136e/worker/games.py#L197, so it would be interesting to know whether it cleanly returns from run_game and if so, why it breaks out of the while loop.

[ppigazzini]

@ddugovic perhaps you are still using an old cutechess-cli that needs libqtcore4. Check this hypothesis with sudo ldd /home/fishtest/fishtest/worker/testing/cutechess-cli or, more simple, delete all the content from the folder sudo rm -rf /home/fishtest/fishtest/worker/testing/* and restart fishtest.

[ddugovic]

My cutechess-cli (using QT5) is still running; results.pgn currently ends in:

[Event "?"]
[Site "?"]
[Date "2017.10.27"]
[Round "105"]
[White "stockfish"]
[Black "base"]
[Result "0-1"]
[FEN "rnbqkb1r/pp1ppppp/5n2/2p5/8/2PPP3/PP3PPP/RNBQKBNR b KQkq - 3+3 0 1"]
[PlyCount "72"]
[SetUp "1"]
[Termination "adjudication"]
[TimeControl "12.33+0.12"]
[Variant "3check"]

1... Nc6 {+0.89/15 0.74s} 2. d4 {-0.98/14 0.43s} d5 {+1.10/15 0.59s}
3. Be2 {-1.02/15 1.1s} e6 {+1.57/13 0.13s} 4. f4 {-0.93/14 0.47s}
Bd6 {+0.75/15 0.72s} 5. Nf3 {-0.60/14 0.46s} O-O {+0.56/14 0.73s}
6. O-O {-0.56/12 0.053s} Ne7 {+0.57/14 0.78s} 7. Ne5 {-0.18/13 0.27s}
Nf5 {+0.28/13 0.46s} 8. g4 {-0.31/14 0.43s} Nh4 {+0.71/14 0.15s}
9. Nd2 {-0.62/15 1.0s} Ne4 {+0.91/17 1.2s} 10. Kh1 {-0.29/14 0.17s}
Bxe5 {+0.57/17 0.49s} 11. Nxe4 {-0.73/16 0.71s} dxe4 {+0.69/14 0.046s}
12. dxe5 {-0.49/13 0.089s} Bd7 {+0.76/15 0.27s} 13. b4 {-0.35/15 0.35s}
cxb4 {+1.11/16 0.26s} 14. cxb4 {-1.03/17 1.1s} Rc8 {+0.95/17 1.5s}
15. a4 {-1.20/17 1.2s} Bc6 {+1.37/16 0.46s} 16. b5 {-1.20/16 0.49s}
Bd5 {+1.45/15 0.054s} 17. Bb2 {-1.90/18 1.6s} Nf3 {+2.33/17 0.65s}
18. g5 {-1.67/17 0.21s} Bc4 {+2.26/19 0.56s} 19. Bxc4 {-2.35/15 0.073s}
Rxc4 {+2.45/15 0.047s} 20. Qe2 {-2.03/18 0.22s} Qa5 {+2.46/17 0.26s}
21. Rac1 {-2.46/20 0.65s} Rxc1 {+2.50/19 0.53s} 22. Rxc1 {-2.59/17 0.18s}
Qxa4 {+2.49/16 0.060s} 23. Qc2 {-2.82/19 1.0s} Qxc2 {+3.13/17 0.17s}
24. Rxc2 {-2.95/20 0.62s} Rd8 {+3.15/15 0.052s} 25. Bd4 {-3.52/21 1.3s}
g6 {+3.45/17 0.34s} 26. Rc4 {-3.85/19 0.80s} Kg7 {+3.56/17 0.19s}
27. h3 {-4.00/17 0.34s} Rd7 {+3.82/17 0.28s} 28. Rc2 {-4.57/16 0.12s}
Nxd4 {+6.01/17 0.38s} 29. Rd2 {-4.85/16 0.12s} Rc7 {+24.47/16 0.27s}
30. exd4 {-16.67/14 0.12s} Rc1+ {+M17/20 0.20s} 31. Kh2 {-M16/19 0.12s}
Re1 {+M15/22 0.046s} 32. f5 {-M14/20 0.10s} gxf5 {+M13/27 0.20s}
33. Rg2 {-M12/23 0.10s} e3 {+M11/29 0.049s} 34. g6 {-M10/24 0.019s}
e2 {+M9/38 0.21s} 35. gxf7+ {-M8/32 0.11s} Kxf7 {+M7/49 0.19s}
36. Rg7+ {-M6/35 0.025s} Kxg7 {+M5/97 0.19s}
37. h4 {-M4/87 0.12s, Black wins by adjudication} 0-1

[ianfab]

@ddugovic
Could you explain how you use priority of tests? If there are only very few tests in the queue it usually does not make a difference, but I would still like to clarify. For me the interpretation of priority values is as follows: 0 - treat it like any other incoming test; <0 - not very important or waiting for other tests to finish, only process if queue is otherwise empty or until priority is increased; >0 - very important/urgent, prioritize over all other existing and newly incoming tests until this is finished.

If it is just about prioritizing one test over another, I would suggest to decrease the priority of the less interesting/important patch instead of increasing the priority of the other, or you can also change the throughput and keep the priority the same. So far I only use positive priorities if there is a test that has to finish quickly (e.g., an urgent bugfix) or if an SPSA tuning is almost finished (e.g., only 200 of 10000 games left) and would otherwise remain in the queue for a long time instead of just quickly finishing the last few games in order to then test the tuned values.

[ddugovic]

@ianfab OK. In cases where I'm available to quickly respond to a test result (especially after a long and difficult coding session where I had many ideas but couldn't write down all of them) I tend to set its priority to +1 or very rarely +2.

And if I've submitted an SPSA which seems finished, or a test which I expect to pass by a wide margin & it isn't passing (but looks like it will take a long time to fail), I reduce its priority to 0 or lower.

I suppose in lieu of increasing a task priority, if I want immediate test results I can (disable my fishnet client and) run tests offline with different parameters & less computing power.

[ianfab]

In cases where I'm available to quickly respond to a test result (especially after a long and difficult coding session where I had many ideas but couldn't write down all of them) I tend to set its priority to +1 or very rarely +2.

That's understandable, but it is an approach that can not work if more than one developer uses it. Blocking all other tests in favor of a single test might discourage other developers, although we unfortunately rarely have the "problem" of too many developers contributing tests, but rather that the workers run out of tasks.

I suppose in lieu of increasing a task priority, if I want immediate test results I can (disable my fishnet client and) run tests offline with different parameters & less computing power.

I do not want to discourage you from using fishtest resources for your tests. What would be a case where you would like to see immediate results in order to react to it, and how? In my experience it is more effective to rather try many different ideas and do not care too much about a single one, and it can also always be picked up at a later point. And if I do not have time to write down the code for an idea and want to avoid to forget it, I just take some notes. At least for me it's always nice to have a list of good ideas that I can pick up at times when I am less creative.

[ddugovic]

I agree that some communication/coordination is necessary; my intent isn't to block other tests but simply to include it among higher-priority tests (other developers can do the same, fishtest resources are through others' generosity so I won't object).

I also agree that in many cases trying multiple ideas makes sense, as opposed to trying complex ideas or ideas which build upon one another.

One recent case where having immediate (or high priority) results is useful is with atomic chess issue #43 which I locally tried many ideas (with various bench simulations, most of which visually looked unpromising) and published two commits to GitHub (gives_atomic_check in particular took many coding attempts to be readable, performant, and accurate):
https://github.com/ddugovic/Stockfish/commit/8cfe23edf4d942d41cfdae440fe89b5d42b1f6d5
https://github.com/ddugovic/Stockfish/commit/0eaa427cb4247c1c0982dde626a0e0bbc9725150
and I had incorrectly assumed http://35.161.250.236:6543/tests/view/5a3ad9426e23db35f28bb05f would fail so I started coding another patch combining those two ideas. If that combined patch failed, I'd probably see if there's any way to salvage atomic_gives_check to test yellow or better.

(Actually, I had written the combined patch first then realized that I need to test the second idea - atomic SEE for evasions - independently, otherwise a simplification might remove the atomic_gives_check patch. But having atomic SEE for evasions could significantly affect atomic move-ordering & searches so there is some merit to rebasing atomic_gives_check once the atomic SEE move-ordering patch passes & is merged.)

I expect http://35.161.250.236:6543/tests/view/5a3b36a96e23db35f28bb069 will pass. If it's red I might try another 1-priority test to understand how atomic SEE loses Elo; if yellow I'll try as you suggest (multiple 0-priority tests since atomic SEE doesn't lose Elo).

I am impressed that http://35.161.250.236:6543/tests/view/5a3a67916e23db35f28bb05b is yellow at STC! It's a complicated patch so I expected it would quickly fail and I'd need to start over. Now that I have some confidence in the key idea (atomic_gives_check) not losing Elo, I can submit 0-priority [0, 10] tests until I find a solution which gains enough Elo to justify the added complexity.

[ianfab]

I agree that some communication/coordination is necessary; my intent isn't to block other tests but simply to include it among higher-priority tests (other developers can do the same, fishtest resources are through others' generosity so I won't object).

But that does not really make a difference compared to using priority 0, it just shifts the commonly used value by 1. I prefer to use a negative value for low-priority tasks and keep 0 for all normal tasks. I do not want to judge the priority of tests from other developers, so all tests should start with 0 unless a developer thinks that the priority of one of his tests can be lowered (which I often do when I just want to measure how much Elo a certain part of the code is worth or when I am filling the queue with speculative LTC tests or tuning sessions until I have time to test more creative ideas).

Regarding http://35.161.250.236:6543/tests/view/5a3a67916e23db35f28bb05b I think that the situation that a capture discovers a check (and does not have the best SEE) is rather rare and the non-capture code in gives_atomic_check seems to be unused, so the effect should be relatively small, but the patch might still perform well if it efficiently helps to avoid some blunders.

[ddugovic]

I do not want to judge the priority of tests from other developers, so all tests should start with 0 unless a developer thinks that the priority of one of his tests can be lowered

Fair enough; the fishtest resources are possible through others' generosity so I won't contest this policy.

[ddugovic]

Regarding http://35.161.250.236:6543/tests/view/5a3a67916e23db35f28bb05b I think that the situation that a capture discovers a check (and does not have the best SEE) is rather rare and the non-capture code in gives_atomic_check seems to be unused, so the effect should be relatively small, but the patch might still perform well if it efficiently helps to avoid some blunders.

I fully agree. May I therefore submit that patch as an inverse simplification at LTC (and if successful, merge it into master)?

[ianfab]

Just because it might help to avoid some blunders does not justify to use different bounds, in my opinion, especially since it adds a lot of code. Making an exception always comes at the cost of having to answer why it was made in one case and not in the other, so there need to be good reasons. If I remember correctly, adding indirect king attacks to quiet checks in qsearch (http://35.161.250.236:6543/tests/view/5899edfe6e23db014ac09fe5) helped a lot to stabilize search at low depths, but I think we should search for a solution that really improves playing strength and not just looks good in analysis. I could maybe retry that patch to see if it works better now, almost a year later. Edit: I apparently already retried it a few months ago with the same result (yellow), see http://35.161.250.236:6543/tests/view/59aaa7f56e23db64939d6742.

[ianfab]

@ppigazzini @Nordlandia After the update of the worker version, your workers go offline. Please let me know if there are issues or whether it is just a coincidence. On my workers the transition worked smoothly, so I do not know what could be reason.

[ianfab]

They are coming back online now, sorry for the noise.

[ppigazzini]

@ianfab I restarted manually my workers.

• w/ WSL the worker always stops during an update
• w/ linux I run the worker using systemd (to switch cleanly between variant & official) and redirecting the outputs to /dev/null

[alwey]

Better testing with or without contempt?

[ianfab]

That is a good question, but since there is an ongoing discussion upstream, I think we should wait for test results and decisions there, and then we can still do further testing if we think that is required or just take over the decision. The draw rate is low in most variants, so it will probably anyway not have a big effect.

[ddugovic]

@ianfab I propose that parameter tweaks can be submitted as inverse simplifications (-5, 10) instead of the standard (0, 10). This would encourage new developers to submit parameter tweaks, and be more likely to gain Elo than to lose Elo. I'd even suggest this makes sense upstream, although @official-stockfish development is already quite noisy, so who knows what they'd say.

I'm looking at http://35.161.250.236:6543/tests/view/5a7b57446e23db0fbab0d3ba which passed STC and has played 9,000 LTC games with a plus score:

LLR: -1.24 (-2.94,2.94) [0.00,10.00]
Total: 9044 W: 2432 L: 2360 D: 4252

[ianfab]

Parameter tweaks probably should have less strict bounds, but I am unsure which ones to use. I have tried [0,8] before, but tests seem to take too long to finish. I think that a couple of aspects have to be considered when choosing the bounds:

• false positive rate
• false negative rate
• average number of games
• a priori Elo distribution of tests that are submitted (relevant for the compromise regarding false positives/negatives)
• risk of overfitting

Below are some numbers to start with. I currently do not have the time to think much about this, but I will try to come back to this topic in a few days.

./sprt -10 10 1 20000 250 -5 10
BayesElo    Elo %Pass   Avg
-10.00  -6.20   0.0067  1876
-9.00   -5.58   0.0100  2034
-8.00   -4.96   0.0154  2208
-7.00   -4.34   0.0226  2404
-6.00   -3.72   0.0319  2640
-5.00   -3.10   0.0484  2886
-4.00   -2.48   0.0708  3159
-3.00   -1.86   0.1040  3455
-2.00   -1.24   0.1435  3795
-1.00   -0.62   0.2026  4075
0.00    -0.00   0.2708  4379
1.00    0.62    0.3573  4568
2.00    1.24    0.4516  4742
3.00    1.86    0.5503  4711
4.00    2.48    0.6468  4623
5.00    3.10    0.7294  4384
6.00    3.72    0.8005  4090
7.00    4.34    0.8558  3766
8.00    4.96    0.8996  3466
9.00    5.58    0.9299  3170
10.00   6.20    0.9536  2892

./sprt -10 10 1 20000 250 0 10
BayesElo    Elo %Pass   Avg
-10.00  -6.20   0.0003  2379
-9.00   -5.58   0.0002  2554
-8.00   -4.96   0.0007  2749
-7.00   -4.34   0.0007  2977
-6.00   -3.72   0.0017  3237
-5.00   -3.10   0.0026  3559
-4.00   -2.48   0.0045  3944
-3.00   -1.86   0.0092  4389
-2.00   -1.24   0.0152  4945
-1.00   -0.62   0.0291  5594
0.00    -0.00   0.0467  6449
1.00    0.62    0.0828  7424
2.00    1.24    0.1449  8435
3.00    1.86    0.2349  9413
4.00    2.48    0.3550  10264
5.00    3.10    0.4990  10540
6.00    3.72    0.6417  10255
7.00    4.34    0.7619  9460
8.00    4.96    0.8509  8435
9.00    5.58    0.9111  7407
10.00   6.20    0.9492  6460

[ddugovic]

I have merged 3f1eb85a1ceb1b408f8f51cb82064b69e095399d which fixes an upstream bug that may have affected recent tests.

[ianfab]

@ddugovic Something with the regression test you submitted seems to be wrong, since the workers can not run any games on it:

Exception running games:
Traceback (most recent call last):
  File "fishtest-master/worker/worker.py", line 88, in worker
  File "/home/fishtest/part_1/fishtest-master/worker/games.py", line 366, in run_games
    setup_engine(new_engine, worker_dir, run['args']['resolved_new'], repo_url, worker_info['concurrency'])
  File "/home/fishtest/part_1/fishtest-master/worker/games.py", line 116, in setup_engine
    f.write(requests.get(github_api(repo_url) + '/zipball/' + sha, timeout=HTTP_TIMEOUT).content)
  File "/home/fishtest/part_1/fishtest-master/worker/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/home/fishtest/part_1/fishtest-master/worker/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/fishtest/part_1/fishtest-master/worker/requests/sessions.py", line 276, in request
    prep = req.prepare()
  File "/home/fishtest/part_1/fishtest-master/worker/requests/models.py", line 221, in prepare
    p.prepare_url(self.url, self.params)
  File "/home/fishtest/part_1/fishtest-master/worker/requests/models.py", line 287, in prepare_url
    raise MissingSchema("Invalid URL %r: No schema supplied" % url)
MissingSchema: Invalid URL u'\thttps://api.github.com/repos/niklasf/Stockfish/zipball/1a832543e4e8ba21f33a5a66519d10ff5aa02eaa': No schema supplied
Task exited

[ddugovic]

@ianfab Oops, I was trying to run a regression test like the ones you submitted. Thanks.

[ianfab]

@ddugovic Your worker seems to produce strongly biased results for shatranj compared to the other workers. I think I did not force a worker update since the shatranj opening book was changed, so it might be that your worker still uses the old one. Can you please check?

[ddugovic]

@ianfab I see that although it did pull changes, perhaps it hadn't restarted? So I restarted it after:

$ git pull origin master
From https://github.com/ianfab/fishtest
 * branch            master     -> FETCH_HEAD
Already up to date.

Something strange is occurring:

Apr 01 07:17:27 dugovic-host systemd[1]: Started Job that runs the fishtest worker daemon.
Apr 01 07:17:36 dugovic-host python[26658]: Warning: 2 opening repetitions vs 1 games per encounter
Apr 01 07:17:36 dugovic-host python[26658]: Warning: Invalid FEN string: rnbkq2r/pppppppp/3b1n2/8/8/7N/PPPPPPPP/RNBKQBR1 w rnbkq2r/pppppppp/3b1n2/8/8/7P/PPPPPPPR/RNBKQBN1 w
Apr 01 07:17:36 dugovic-host python[26658]: Warning: The opening suite is incompatible with the current chess variant
Apr 01 07:17:37 dugovic-host python[26658]: Worker version 68 connecting to http://35.161.250.236:6543
Apr 01 07:17:37 dugovic-host python[26658]: Verifying signature of stockfish_5d604e1a5d5501e48a7128168f8daa33de985356 ...
Apr 01 07:17:37 dugovic-host python[26658]: Verifying signature of stockfish_61a0455b6fd49c7574fbc9b02456c2bc6d71f473 ...
Apr 01 07:17:37 dugovic-host python[26658]: CPU factor : 3.149501 - tc adjusted to 94.49+0.94
Apr 01 07:17:37 dugovic-host python[26658]: Running master vs shatranj-master

I set my worker/fishtest.cfg to use a BMI2 architecture, using gcc version 8.2.0:

make_cmd = 'make -j3 build ARCH=x86-64-bmi2 COMP=gcc'

[ianfab]

@ddugovic The book is downloaded from a different repository than the worker code itself. You just need to delete the shatranj.epd from the working directory of the worker, then it should automatically download the current version of the file (as soon as it is required).

[ianfab]

@ddugovic The results look much more consistent now, so thanks for fixing it.

[ianfab]

@ppigazzini @ddugovic I am starting to test the large-board version of my fairy-SF fork on the variant fishtest server. If you observe any issues please let me know (and if so, sorry in advance). On my worker it seems to work fine so far.