Leela-chess: First play urgency

[jkiliani]

I just rather naively ported the modified first play urgency reduction from Leela Zero (https://github.com/gcp/leela-zero/commit/47c06a8d87809fd9b600fac67e979223c0a46221, https://github.com/gcp/leela-zero/commit/6b56e3ffbeb513e643e144397a4421e53c1879e5, discussions in https://github.com/gcp/leela-zero/issues/696, https://github.com/gcp/leela-zero/pull/827, https://github.com/gcp/leela-zero/pull/866). I'm now running a tournament of the modified binary against master with cd1a1e:

./cutechess-cli -rounds 100 -tournament gauntlet -concurrency 2 -pgnout results.pgn \
 -engine name=lc_gen13_fpu cmd=lczero_fpu arg="--threads=1" arg="--weights=$WDR/cd1a1e" arg="--playouts=800" arg="--noponder" arg="--noise" tc=inf \
 -engine name=lc_gen13 cmd=lczero arg="--threads=1" arg="--weights=$WDR/cd1a1e" arg="--playouts=800" arg="--noponder" arg="--noise" tc=inf \
 -each proto=uci

It looks like it might potentially already give a small benefit, but too early to tell really. At least it seems unlikely I bugged the binary with the port, or the FPU reduction should clearly lose. Current standings:

Score of lc_gen13_fpu vs lc_gen13: 7 - 5 - 2  [0.571] 14

I'll report the result when it's finished, but from the experience with Leela Zero, FPU reduction works better the stronger the neural net is, so maybe it's still too early for it to give a benefit right now.

[jkiliani]

A short recap of what FPU reduction does: It makes the evaluation of nodes that have not been visited yet worse by a flat reduction. This results in less exploration of unvisited and often bad nodes, while significantly increasing the exploitation of visited nodes, and in consequence, the search depth. I think the effect is actually similar to the pruning techniques used by Alpha-Beta chess engines. In @gcp's tests with Leela Zero on strong neural nets, this technique gives an improvement of ~200 Elo. All credit goes to @Eddh who came up with the concept.

My tournament now strongly indicates FPU reduction already helps considerably:

Score of lc_gen13_fpu vs lc_gen13: 21 - 7 - 5  [0.712] 33

About the implementation, I have to apologise that I'm not experienced with maintaining branches on Github. The main change is in UCTNode::uct_select_child in UCTNode.cpp:

UCTNode* UCTNode::uct_select_child(Color color) {
    UCTNode* best = nullptr;
    float best_value = -1000.0f;

    LOCK(m_nodemutex, lock);
    // Count parentvisits.
    // We do this manually to avoid issues with transpositions.
    auto total_visited_policy = 0.0f;
    auto parentvisits = size_t{0};
    for (const auto& child : m_children) {
        parentvisits += child->get_visits();
        if (child->get_visits() > 0) {
            total_visited_policy += child->get_score();
        }
    }
    auto numerator = static_cast<float>(std::sqrt((double)parentvisits));
    auto fpu_reduction = cfg_fpu_reduction * std::sqrt(total_visited_policy);

    for (const auto& child : m_children) {
        // get_eval() will automatically set first-play-urgency
        auto winrate = child->get_eval(color);
        if (child->get_visits() == 0) {
            // First play urgency
            winrate -= fpu_reduction;
        }
        auto psa = child->get_score();
        auto denom = 1.0f + child->get_visits();
        auto puct = cfg_puct * psa * (numerator / denom);
        auto value = winrate + puct;
        assert(value > -1000.0f);

        if (value > best_value) {
            best_value = value;
            best = child.get();
        }
    }

    assert(best != nullptr);
    return best;
}

Apart from this, I simply added cfg_fpu_reduction in Parameters.h, and set the values for both this and cfg_puct in Parameters.cpp to the latest values from @gcp's tuning run for Leela Zero:

    cfg_puct = 0.8f;
    cfg_fpu_reduction = 0.25f;

If @glinscott or @Error323 are interested, one of you could create a pull request so more people can test this out under different settings?

[Uriopass]

It seems like it reduces exploration to win some ELO, but that's not really what we want during self-play, do we ? Or is it just for tournaments ?

[jkiliani]

No, this is also for training. There was an extensive debate in Leela Zero whether this would hurt training, or the effectiveness of Dirichlet noise etc., see the links in my first post. @gcp eventually decided to pull this on the assumption that it creates stronger training data too. In hindsight there was nothing wrong with that decision, since Leela Zero got way stronger since the merge.

[remdu]

Alderi did some tests on 9x9 go though and there wasn't much difference with normal fpu (parent's net_eval) in terms of network quality over time. And ultimately there are a lot of questions left, such as what are the optimal values, or is there a better formula for fpu ? There is potential with variable fpu based on the average value of child nodes or the Q value of parent node but I've never been able to see something consistantly better than current fpu reduction. It's also up for debate if using sqrt(total_visited_policy) is really better than raw total_visited_policy.

[jkiliani]

I agree, but at this point the Leela Chess networks are still relatively weak, so I gave the modified FPU reduction the best shot at already being an improvement. I'm actually surprised it already works this well... since you are here, do you see any problems in how I ported it?

[remdu]

It seems fine.
I agree that it will be interesting to have this in LCZero, since it's faster than LeelaZero to play games right ? Potentially we can learn more about these kinds of things in LCZero since testing is cheaper.

[jkiliani]

Thanks for reviewing. Yes, LCZero is currently much faster, since

1. Effectively 800 visits (since NNCache without tree reuse) instead of LeelaZero's 3200 visits
2. 64x6 neural net instead of 128x10
3. Smaller board results in faster convolutions of the residual layers

The network size will eventually have to be raised, I'm hoping for 128 filters and 10-12 blocks, but the 800 visits can probably be kept indefinitely because of the lower branching factor, so it should always stay faster than LeelaZero.

[Error323]

Interesting. I think with the project now well underway we should maintain a next branch as well. And treat master as stable with only tagged versions and bugfixes. Then we can merge this into next IMHO.

[gcp]

It makes the evaluation of nodes that have not been visited yet worse by a flat reduction.

That's not what the current implementation does. I'll link to the corresponding commit:

https://github.com/gcp/leela-zero/commit/6b56e3ffbeb513e643e144397a4421e53c1879e5

I think the description will give you an "aha" moment as to why it works well.

It seems like it reduces exploration to win some ELO, but that's not really what we want during self-play, do we?

It's an open question, right? If you want exploration, you can increase the softmax temperature on the policy output, increase the noise, or increase the UCT constant. There's probably some optimum, but finding it is going to be hard. But there's little reason to worry only bout FPU influence only in this regard.

[CMCanavessi]

I think with the project now well underway we should maintain a next branch as well. And treat master as stable with only tagged versions and bugfixes. Then we can merge this into next IMHO.

That would be cool. An official release that you can package (like I've been doing with the latest builds) that would be the "real" released version and could be used to somehow standarize testing, instead of every one of us using different dev versions. Kinda like SF's model where you have a stable usable version number and then you may or may not want to use the gazillion dev versions there are out there.

[jkiliani]

Results are in:

Score of lc_gen13_fpu vs lc_gen13: 65 - 22 - 13  [0.715] 100
Elo difference: 159.78 +/- 70.53

As I mentioned before, this is much better than I expected for a net of this strength. Judging by Leela-Zero's experiences, the margin of difference may be even larger once the neural net policy priors are better, but that is a question for another day. I'm happy this turned out so well.

How do we go from here? I would recommend that once there is a pull request, we should test at a few more networks, and maybe different playout counts as well. Also probably a good idea would be to rerun @killerducky's analysis in https://github.com/glinscott/leela-chess/issues/105 to test how well the feedback loop for moves found by Dirichlet noise works with FPU reduction. If everything checks out, we gain 150-200 Elo from this improvement in search efficiency.

@Uriopass linked one of the games below. I put the complete match records here: https://drive.google.com/open?id=1DMHlNB6OrD2fb09Oj3sMS4b53n7AEVfW

[Uriopass]

White uses fpu

[jkiliani]

I'll delete the first of the two games from my post, could you tell me how you embed the .pgn player in your post?

[Uriopass]

I don't embed anything, I use http://www.apronus.com/chess/wbeditor.php to turn a pgn into a .gif and drag it to github :)

[jkiliani]

Thanks, I'll try that next time. For now, one sample should be fine, I put the rest on Googledrive if people are interested.

[sf-x]

So it's an analog of late move reduction in alpha-beta. In future, how about "reducing" later moves more, as in many LMR implementations?

[remdu]

Yeah You can try that. The current fpu method in LeelaZero works but I think there are a lot of possibilities. In practice I found many solutions that are similar in strength, but I couldn't see any improvement consistent among different networks. Maybe I could detect an improvement with more test games. I ran 150 test games at most. Anyway you can try a lot of things. I haven't found any existing research about fpu methods, it's usually just a constant or you are supposed to explore every move first.
From my experience I think there are 2 main aspects to fpu : it is better if it tracks the winrate you can reasonably expect from an unexplored node, and then it should be slightly lower than that, probably because playouts are then concentrated into the few reasonable moves the net sees and can do more precise choice between these few.

[glinscott]

@jkiliani wow, that's a huge strength increase! Definitely agree we want to be careful, and validate that it's stronger on the next promoted net as well let's say, but this looks incredibly promising.

[jkiliani]

Agreed, should be tested thoroughly. @Error323 recommended starting a next branch similar to Leela Zero, so changes like this can be fully evaluated before merging them to master.

[killerducky]

~@jkiliani can you implement the current LZGo version with sqrt(total_visited_policy)?~

Oops you did!

[crocogoat]

Could this potentially hurt in the long term? for a few "bad" nodes aren't really bad, for example sacrifices. SF reduces all kinds of pruning with rules, so will figure things out often enough, but some positions are harder and take much longer. Is there the risk that LCZ would be "staring blind" on "good" moves like SF sometimes does?

I do think SF has generally proven that it's ok to prune _a lot_ and that such positions in high-quality chess games are rare enough to not really matter, but would it be detrimental to the ultimate potential strength of LCZ considering the learning method?

[jkiliani]

This whole discussion was already done regarding Leela Zero, where FPU reduction was implemented first. From the experience of Leela Zero, it worked and we did not notice any long term drawbacks, after having it active for over a month. I linked various threads in the first post for this purpose.

[remdu]

There are also solutions which don't give an elo increase as drastic but that are probably safer for self play and don't require magic numbers. I think using the average q value of child nodes as fpu could be good for that purpose.

[jkiliani]

I would suggest we just pool ideas for a while, and then evaluate the relative advantages of the various proposed solutions. Maybe there are alternatives that offer comparable strength boosts after all? This was just a starting point for a discussion to show the huge potential for improvement in the MCTS algorithm.

In my opinion, modified FPU reduction as proposed in this thread shouldn't do any harm since it doesn't appear LZ experienced any ill effects from the implementation. As it has been proven to give a big boost, alternatives should ideally compared favourably to it. I agree that the magic number is a bit of a drawback.

[jkiliani]

I just ran @killerducky's debug procedure https://github.com/glinscott/leela-chess/issues/106#issuecomment-373256679 with the gen5 net on the low policy mate-in-one position:

position startpos moves e2e4 f7f6 d2d4 d7d6 c2c4 a7a5 b1c3 e7e5 g1e2 c8e6 d4e5 c7c6 e5f6 h7h5 c3d5 e6d5 d1d5 c6d5 b2b4 d5e4 b4a5 a8a5 c1h6 g8f6 h2h3 b8c6 a1b1 b7b5 b1b5 c6d4 a2a4 d4b5 h6f4 h5h4 e1d2 f6d5 e2c1 d8d7 a4b5 g7g6 f4h6 d5e7 h1g1 e7f5 c1a2 e8e7 g2g4 f5h6 a2b4 d6d5 b4d5 e7e6 d2e1 d7a7 f1g2 a7a8 d5f6 e6f6 g4g5 f6g5 g1h1 a8b8 h1f1 b8b6 f1g1 f8b4 e1d1 g5f6 d1c2 a5a3 g2f1 h6f5 f1e2 a3h3 g1a1 b4a5 c2b2 b6e3 a1a5 e3d2 b2a1 g6g5 a5a3 h3g3 a3a8 h8a8 a1b1 a8a4 f2f3 a4b4 b1a1
go

Here are 20 runs of this position with the master branch, withlczero -w ~/Downloads/LCZero-Networks/gen5-64x6.txt -p 800 --noponder -n -l log2.txt

d2b2 ->     107 (V: 100.00%) (N:  1.15%) PV: d2b2 
d2b2 ->     106 (V: 100.00%) (N:  1.81%) PV: d2b2 
d2b2 ->      90 (V: 100.00%) (N:  1.38%) PV: d2b2 
d2b2 ->     130 (V: 100.00%) (N:  1.99%) PV: d2b2 
d2b2 ->     118 (V: 100.00%) (N:  1.26%) PV: d2b2 
d2b2 ->      98 (V: 100.00%) (N:  1.32%) PV: d2b2 
d2b2 ->      75 (V: 100.00%) (N:  0.84%) PV: d2b2 
d2b2 ->      90 (V: 100.00%) (N:  0.86%) PV: d2b2 
d2b2 ->      90 (V: 100.00%) (N:  1.08%) PV: d2b2 
d2b2 ->     139 (V: 100.00%) (N:  2.01%) PV: d2b2 
d2b2 ->     146 (V: 100.00%) (N:  2.30%) PV: d2b2 
d2b2 ->     112 (V: 100.00%) (N:  1.06%) PV: d2b2 
d2b2 ->      98 (V: 100.00%) (N:  0.94%) PV: d2b2 
d2b2 ->     104 (V: 100.00%) (N:  1.01%) PV: d2b2 
d2b2 ->      85 (V: 100.00%) (N:  0.81%) PV: d2b2 
d2b2 ->      81 (V: 100.00%) (N:  0.81%) PV: d2b2 
d2b2 ->      99 (V: 100.00%) (N:  0.92%) PV: d2b2 
d2b2 ->      91 (V: 100.00%) (N:  0.81%) PV: d2b2 
d2b2 ->      77 (V: 100.00%) (N:  0.90%) PV: d2b2 
d2b2 ->     157 (V: 100.00%) (N:  2.53%) PV: d2b2 

The mate is found every time, as soon as it get one visit the search converges on it. Now with FPU reduction:

d2b2 ->     120 (V: 100.00%) (N:  4.60%) PV: d2b2 
d2b2 ->      52 (V: 100.00%) (N:  1.81%) PV: d2b2 
d2b2 ->      37 (V: 100.00%) (N:  1.19%) PV: d2b2 
d2b2 ->      65 (V: 100.00%) (N:  2.40%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.83%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.84%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.81%) PV: d2b2 
d2b2 ->     118 (V: 100.00%) (N:  4.70%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.81%) PV: d2b2 
d2b2 ->      62 (V: 100.00%) (N:  2.06%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.82%) PV: d2b2 
d2b2 ->      47 (V: 100.00%) (N:  1.48%) PV: d2b2 
d2b2 ->      38 (V: 100.00%) (N:  1.24%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.82%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.93%) PV: d2b2 
d2b2 ->      45 (V: 100.00%) (N:  1.44%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.81%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.81%) PV: d2b2 
d2b2 ->       0 (V: 98.58%) (N:  0.81%) PV: d2b2 
d2b2 ->      92 (V: 100.00%) (N:  3.43%) PV: d2b2 

Here the mate is completely missed when Dirichlet noise doesn't help raise the policy prior. We should analyse this some more, maybe exempting the root node from FPU reduction would be a good idea...

It seems that for a very low policy move, Dirichlet noise needs to raise the prior to ~1.15% in order to be found at 800 playouts with FPU reduction, while without it, moves as low as ~0.45% are found at 800 playouts. I think the feedback loop would still work, but it may be worth thinking about solutions for getting the root priors of low-policy, high-value moves up enough to ensure they are found and make it to the training data.

[jkiliani]

Here is the match result for the gen14 network:

Score of lc_gen14_fpu vs lc_gen14: 65 - 29 - 6  [0.680] 100
Elo difference: 130.94 +/- 71.72

A bit closer but still not bad. I just started the tournament for gen15, will probably continue for one or two nets after that so we have some statistics that this (hopefully) works reliably for all future networks.

My proposition to address the concern I wrote of above (finding high-value, low-policy moves reliably enough to make the training feedback loop work) is the following, that I would unfortunately need some help with coding: I was thinking of disabling pruning i.e. FPU reduction for the root only, in games that the engine knows are training games (how about when --randomize is set?). That way, any moves at the root node that Dirichlet noise helps at least a little should end up with one visit, which would definitely trigger others if the node turned out to be good. Would maybe @Eddh or @killerducky know how we could disable FPU reduction for the root node only? What do other people here think about this proposition?

[jkiliani]

When FPU reduction was merged to master on Leela Zero, we had a number of network matches that contained both v0.11 clients (without FPU reduction) and v0.12 client with FPU reduction: https://github.com/gcp/leela-zero/issues/78#issuecomment-367328001

For a number of the tested networks, results differed considerably between the two client versions, which indicates that FPU reduction does actually work better with some networks than others. It looks now as if the same holds true for Leela Chess. With 6a5ccd, FPU reduction didn't do that great:

Score of lc_gen15_fpu vs lc_gen15: 58 - 36 - 6  [0.610] 100
Elo difference: 77.71 +/- 68.47

I have yet to see any network where it's actually worse, but the margin of improvement seems to differ. I'll probably have to revise my estimation of the gain downward a bit. Of course it's also distinctly possible that cd1a1e was somewhat lucky, since 100 games really isn't that good a statistic, just the best I can manage on my hardware. In any case I should probably test like 10 or so networks in this manner, so we get at least a general idea how variable the improvement from FPU reduction actually is.

[rohitrajiit]

cannot neural network for search learn this on its own

[gcp]

cannot neural network for search learn this on its own

FPU tells the search what to do for positions where the neural network hasn't run yet. You have to put something there.

[Ishinoshita]

@jkilliani 2d ago

This whole discussion was already done regarding Leela Zero, where FPU reduction was implemented first. From the experience of Leela Zero, it worked and we did not notice any long term drawbacks, after having it active for over a month.

Kinda games strength vs games diversity dilemma, similar to exploitation vs exploration in tree search. But dilemma does not apply at game playing level but at nnet training level. I.e. would somewhat more diverse games played by somewhat weaker players results in a faster training after a while, namely higher ELO gains after the same number of training cycles ? Settling the question would require more than just trying one of the alternatives. Purely Byzantine comment. Betting on strengh sounds more than reasonable choice ;-) LZ is making so amazing progresses and LCZ will certainly follow the same track. Congats to all of you guys !

[jkiliani]

Score of lc_gen16_fpu vs lc_gen16: 48 - 43 - 9  [0.525] 100
Elo difference: 17.39 +/- 65.72

I think that unfortunately FPU reduction has to be postponed, this is no longer a statistically significant improvement, and the trend over the four matches I did was consistently downward, not up.

I have a hypothesis what may be happening here: As a result from the training, many tactically strong moves are currently being discovered, by are still low on the policy priors. This causes the engine with regular FPU to see them, while with FPU reduction they are often missed. That would mean that while FPU reduction searches deeper, it messes up more often tactically at the moment.

If this hypothesis turns out to be correct, we will eventually see the performance of FPU reduction increase again, once the policy priors on good tactical moves are high enough that they are found even with FPU reduction. I'll monitor this but will probably not test every net anymore, since it now looks like this won't be a candidate for merging anytime soon.

Sorry for getting people's hopes up on this. With LZ we only tested this with a net that was already quite strong. When I tried it here and it seemed to work initially I assumed that gen13 was simply already strong enough for it to work. This was apparently premature.

[jkiliani]

For 38576a, FPU reduction again works much better than the previous net:

Score of lc_gen17_fpu vs lc_gen17: 60 - 32 - 8  [0.640] 100
Elo difference: 99.95 +/- 68.78

Maybe it wasn't a trend after all, but simply a case of very high variance in performance among different nets. The tests I made in https://github.com/glinscott/leela-chess/issues/160#issuecomment-375090149 triggered @gcp to remove FPU reduction on the root node for Leela Zero (https://github.com/gcp/leela-zero/pull/1083). This should definitely also be done for Leela Chess, for merge at a later point, when the strength gain is hopefully more clear. I tried to port his fix but have some trouble at the moment with a call in UCTSearch.cpp that checks whether the current node is root or not (https://github.com/gcp/leela-zero/pull/1083#issuecomment-375797732). This works for Leela Zero but not Leela Chess, anyone have an idea why?

[jhellis3]

It makes sense that this will become more and more effective the stronger the NN becomes. I also like the idea of excluding root moves just to be safe.

[jkiliani]

Surprisingly, best match result so far for FPU reduction, with 74287c:

Score of lc_gen19_fpu vs lc_gen19: 75 - 21 - 4  [0.770] 100
Elo difference: 209.91 +/- 81.17

I may need to get myself a crystal ball to understand this, but it looks like this works well for the majority of nets, and OKish for the rest.

[CMCanavessi]

How does the improvement varies acording to TC ? Is it better with short TC or will it help long TC? Does it varies much?

[jkiliani]

Good question, didn't test this yet. My (speculative) answer is it should help more at long TC, because the effect of pruning on search depth should be more pronounced then. I can run a test later but it would take a long time to finish if I used something like -p 4000.

I posted the code above, you could also use it and run some tests yourself... we just shouldn't use it for self-play yet until the root noise interaction is fixed.

[gcp]

In leela-chess, the m_root is a raw pointer, not a unique_ptr:
UCTNode* m_root;

You can just remove the .get() and it should work.

Not sure when this was cleaned up in Leela Zero, but might've been when tree reuse was added.

[gcp]

One thing to consider wrt the effectiveness of FPU reductions is that one major benefit is that they prevent the search from going full width when the score drops. i.e. if we are at a branch and we were overestimating our position, every expanded node will have a score < parent, and we will explore every successor.

The impact that this has is likely less if you have 35 successors....compared to when you have 361.

[jkiliani]

FPU reduction works fine for Id30 (839e5d), just not as much improvement as 74287c:

Score of lc_id30_fpu vs lc_id30: 56 - 20 - 24  [0.680] 100
Elo difference: 130.94 +/- 62.97

I'll try the noise root fix once more when I have time, it seems the tree reuse port is still bugged.

[jkiliani]

For Id 33, the improvement is not large:

Score of lc_id33_fpu vs lc_id33: 48 - 34 - 18  [0.570] 100
Elo difference: 48.96 +/- 62.77

I now have a version of the code that disables FPU reduction when --randomize is set, so it would be disabled completely for training games. Should do no harm at least in this form, but I'll try to combine this with the tree reuse pull again since I would prefer @gcp's implementation of only deactivating FPU reduction at the noised root.

[jkiliani]

Good improvement for Id 38:

Score of lc_id38_fpu vs lc_id38: 60 - 19 - 21  [0.705] 100
Elo difference: 151.35 +/- 65.62

This is the last test I'm planning to do for different nets. My conclusion from the test series is that FPU reduction does provide a reliable strength boost, although that boost does vary to an extent between networks.

I am now planning to do a cfg_puct tuning run combined with the FPU reduction as I tested here, by way of a round-robin tournament, using the just tested Id 38 as network. Will update when I have results, but this will take a while.

Edit: My tuning run for cfg_puct so far shows a very slight tendency for higher values. cfg_puct = 1.0 seems to be doing best by a slight margin so far, but it's really still too early to tell.

[jkiliani]

Current standing of my cfg_puct tuning tournament, with 250 games finished:

lc_master results: 26 - 60 - 14
lc_fpu_puct07 results: 36 - 42 - 22
lc_fpu_puct08 results: 45 - 38 - 17
lc_fpu_puct09 results: 46 - 36 - 18
lc_fpu_puct10 results: 49 - 26 - 25

This looks like we should probably raise cfg_puct when we introduce FPU reduction. It would likely give a benefit to do so even without it, but I'm not testing that currently.

[jjoshua2]

Is this with more than one net? Which one? It should definitely be tested
on at least 2 or 3 different ones since each net will have a different
optimum.

On Tue, Mar 27, 2018 at 9:24 AM, jkiliani [e-mail] wrote:

Current standing of my cfg_puct tuning tournament, with 189 games finished
so far:

lc_master results: 19 - 45 - 11
lc_fpu_puct07 results: 29 - 30 - 16
lc_fpu_puct08 results: 33 - 29 - 12
lc_fpu_puct09 results: 33 - 27 - 16
lc_fpu_puct10 results: 36 - 19 - 19

This looks like we should probably raise cfg_puct when we introduce FPU
reduction. It would probably give a benefit to do so even without it, but
I'm not testing that currently.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/glinscott/leela-chess/issues/160#issuecomment-376523281,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AO6INKCsFFiCp7dva6x6HDq1JZ4WsqCWks5tij18gaJpZM4SzWmi
.

[jkiliani]

OK I decided to stop my tuning run early, because it already shows actionable results and I think that switching the used net may make this more reliable. With Id 38, the results of the round-robin tournament including the current master branch and FPU reduction with cfg_puct values of 0.7, 0.8, 0.9 and 1.0 after 25 rounds (250 games total):

Rank Name                          Elo     +/-   Games   Score   Draws
   1 lc_fpu_puct10                  81      61     100   61.5%   25.0%
   2 lc_fpu_puct09                  35      63     100   55.0%   18.0%
   3 lc_fpu_puct08                  24      63     100   53.5%   17.0%
   4 lc_fpu_puct07                 -21      61     100   47.0%   22.0%
   5 lc_master                    -123      67     100   33.0%   14.0%

The search code with FPU reduction apparently benefits from a higher cfg_puct than the current value of 0.85. Increased exploration is probably stronger because it reduces the chance of tactical blunders.

I will do another cfg_puct tuning run with the latest net, probably tomorrow. I'll simply split the 1000 games I had originally planned for this into 4 nets so we have some data on variance between networks. I will also remove the worst performing value of 0.7 from the competition and try 1.1 additionally.

[jkiliani]

This run ended a bit earlier than planned since my computer crashed. Nonetheless, results look useful:

lc_master_id45 results: 36 - 37 - 13
lc_puct08_id45 results: 25 - 41 - 19
lc_puct09_id45 results: 37 - 31 - 18
lc_puct10_id45 results: 37 - 32 - 17
lc_puct11_id45 results: 37 - 31 - 17

FPU reduction for this network gives far less benefit than for many others I tested. 0.9, 1.0 and 1.1 all seem to work well for the exploration constant. I'm now going to try Id 48 which failed, because I have a suspicion that it wouldn't have failed with other UCT parameters.

[remdu]

Interesting tests. We didn't try with that many different nets in LZ

[jkiliani]

I know, but the small net size and low playouts for LC0 is the only reason I can do this here. With Leela Zero, a lot more computation would be needed. I raised the topic of cfg_puct there recently but @gcp was not interested since he has tuned this with https://github.com/gcp/leela-zero/commit/6b56e3ffbeb513e643e144397a4421e53c1879e5. The different results for different nets make sense to me since we did see discrepancies between 0.11 and 0.12 clients in several match games when 0.12 was released but not yet enforced.

[jkiliani]

cfg_puct tuning run for Id 48, a failed net:

Rank Name                          Elo     +/-   Games   Score   Draws
   1 lc_puct09_id48                 85      66     100   62.0%   14.0%
   2 lc_puct08_id48                 74      65     100   60.5%   15.0%
   3 lc_puct11_id48                 -3      60     100   49.5%   23.0%
   4 lc_puct10_id48                -74      63     100   39.5%   19.0%
   5 lc_master_id48                -81      66     100   38.5%   13.0%

As I expected, this net would have had no trouble passing with different search parameters. What I found surprising about this run was how poorly cfg_puct = 1.0 did for this network, and how well 0.8 and 0.9. Maybe picking 1.0 isn't optimal after all, I should definitely run this for several more nets.

[jkiliani]

Next tuning run finished, with Id 55:

Rank Name                          Elo     +/-   Games   Score   Draws
   1 lc_puct09_id55                 42      62     100   56.0%   20.0%
   2 lc_puct08_id55                 42      60     100   56.0%   24.0%
   3 lc_puct10_id55                  7      61     100   51.0%   20.0%
   4 lc_puct07_id55                -24      61     100   46.5%   21.0%
   5 lc_master_id55                -67      63     100   40.5%   19.0%

After the last run I thought it unlikely I would need to compare cfg_puct = 1.1 further so I changed it back to 0.7. It now looks like simply leaving cfg_puct at 0.85 (where it currently is now) seems a solid choice, or maybe 0.9 based on the first two tuning runs.

[jkiliani]

With Id 57, FPU reduction is a slight regression:

Rank Name                          Elo     +/-   Games   Score   Draws
   1 lc_master_id57                 28      63     100   54.0%   16.0%
   2 lc_puct09_id57                  7      61     100   51.0%   20.0%
   3 lc_puct10_id57                  3      63     100   50.5%   15.0%
   4 lc_puct08_id57                  3      63     100   50.5%   15.0%
   5 lc_puct07_id57                -42      59     100   44.0%   26.0%

The best cfg_puct value appears to stabilise around 0.9 though. I'm now running Id 62, hopefully after this there's enough data to come to a decision.

[jkiliani]

FPU reduction is a considerable improvement for Id 62:

Rank Name                          Elo     +/-   Games   Score   Draws
   1 lc_puct10_id62                 55      54     120   57.9%   25.8%
   2 lc_puct09_id62                 38      56     120   55.4%   20.8%
   3 lc_puct08_id62                -29      54     120   45.8%   25.0%
   4 lc_master_id62                -64      56     120   40.8%   21.7%

I removed puct = 0.7 from the competition here, since it did not have a realistic shot at becoming best tuning value anymore. I'll take a bit of time to aggregate the results now.

[jkiliani]

And finally, results for Id 65, also only evaluated with cfg_puct = 0.8, 0.9 and 1.0.

Rank Name                          Elo     +/-   Games   Score   Draws
   1 lc_puct09_id65                 53      53     120   57.5%   30.0%
   2 lc_puct10_id65                 50      53     120   57.1%   29.2%
   3 lc_master_id65                -20      56     120   47.1%   19.2%
   4 lc_puct08_id65                -83      56     120   38.3%   23.3%

cfg_puct = 0.9 clearly wins this one too.

Based on tuning runs for seven different nets, the averaged win rates for the different versions were:

master:     43.25 -> -47 Elo
fpu_puct07: 45.83 -> -29 Elo
fpu_puct08: 49.08 -> -6 Elo
fpu_puct09: 55.85 -> 41 Elo
fpu_puct10: 53.24 -> 23 Elo
fpu_puct11: 51.35 -> 9 Elo

Based on these results I propose to change cfg_puct to 0.9, for an average gain of 88 Elo. While a value of 0.95 would very likely work very well too, that was not tested, and 0.9 is the smaller change to the existing constant (0.85). I am updating the pull request #198, this is now ready for merge in my opinion.

[remdu]

Have yoi tried looking at what could be the right value for fpu reduction ?

[jkiliani]

No, I only varied cfg_puct. It's of course a good question if fpu reduction as tuned by @gcp has the correct value here too, but I'd be in over my head trying to tune this: With both fpu reduction and cfg_puct tunable, and multiple networks to tune them for, I'd need thousands of games at least.