Leela-chess: Full temperature setting

[jkiliani]

The temperature parameter for move selection, as far as utilised in Alphago Zero and AlphaZero, is already implemented in LCZero. However, it may be useful to fully implement it as described in the Alphago Zero paper:

They generalise the move probability as the exponentiated visit count, p(n) ~ N^(1/τ) with a temperature parameter τ. For τ=1, this simplifies to move probability proportional to visit count, while for an infinitesimal temperature τ→0 it means greedy selection, i.e. the move with highest visit count is always selected. The two cases τ=1 and τ→0 are currently implemented both in Leela Zero and Leela Chess.

However, there is a good reason for the generalised implementation: τ→0 produces the strongest play, but makes the code deterministic if no other random factors such as Dirichlet noise or symmetries are introduced. For Leela Zero this is not a problem, since the random symmetry applied to every neural net evaluation provides sufficient randomness, but for LCZero this is not an option (yet?). Deterministic code makes evaluation matches impossible since they will end up repeating the same game.

On the other hand, τ=1 has a lot of randomness, but vastly lowers playing strength since even moves that only got a single visit can be selected in the end, usually ending up in blunders that easily turn the game result around.

The advantage of a temperature setting in between, for example τ=0.25, is that it provides non-deterministic behaviour while affecting playing strength much less: Let's assume a scenario where the search found a move A with 400 visits, B with 350, and moves C and D with 40 and 10 visits respectively, and let's say C and D are blunders. With τ=1, A and B are picked with 50% and 44% percent, while a blunder move will be picked with a chance of 6%. If this visit distribution is typical, a blunder will happen at least once a game on average. With τ=0.25 on the other hand, the chances of selection for A, B, C, and D are 63%, 37%, 0.006% and 0.00002%. With τ→0, only A can ever be selected.

So a small but not infinitesimal temperature will select moves with only slightly less visits than the most visited one frequently (which provides the needed randomness), but won't select moves with low visit counts. This would (in my opinion) make it a good choice for implementing in Leela Chess for evaluation games, at least until other methods of providing randomness are found and tested.

The implementation is very easy: In UCTNode::randomize_first_proportionally(), accum has to be changed to a double precision float, and accum += child->get_visits() replaced by accum += child->get_visits()^(1/tau) with tau a parameter specified when enabling cfg_randomize.

Any thoughts?

[glinscott]

This certainly makes sense!

[Error323]

Agreed, it's easy to implement and a nice generalization. It might even be useful for handicapping the engine against humans?

[jkiliani]

For playing against humans, I think it's not so much about a way of handicapping the engine, for that τ=1 or just using a net that's not state of the art seems good enough. I think for people playing lczero, randomness in some form is mostly needed since a predictable playing partner is boring.

For training games, a temperature that decays over the course of the game could also be considered. For the opening, τ=1 is best because it generates a large variety of positions, similar to an opening book. In the endgame, this may hold back the program though, as the quality of the value head depends on any given position being trained with the correct game result, and a won position turned into a loss by a late blunder doesn't do that. The Alphago Zero (and Leela Zero) solution of setting τ=1 for 30 moves and τ→0 for the rest of the game is a simple way to resolve this, but with fractional temperatures implemented, other decay schedules could be tried.

[Error323]

For playing against humans, I think it's not so much about a way of handicapping the engine, for that τ=1 or just using a net that's not state of the art seems good enough. I think for people playing lczero, randomness in some form is mostly needed since a predictable playing partner is boring.

Yeah true, I was thinking in terms of τ>1, but neural nets from different generations is probably more interesting to play against? We'll see once we're well underway :-)

For training games, a temperature that decays over the course of the game could also be considered. For the opening, τ=1 is best because it generates a large variety of positions, similar to an opening book. In the endgame, this may hold back the program though, as the quality of the value head depends on any given position being trained with the correct game result, and a won position turned into a loss by a late blunder doesn't do that. The Alphago Zero (and Leela Zero) solution of setting τ=1 for 30 moves and τ→0 for the rest of the game is a simple way to resolve this, but with fractional temperatures implemented, other decay schedules could be tried.

Indeed. The number 30 seems somewhat arbitrary, a τ that decays as function of depth might be better? It's hard to know without trying though.

[Dorus]

Going to place this here instead of in #105

One possible approach to reduce the endgame bungling would be to not use temperature=1 for the whole game anymore.

Why? Training still happens on the mate in one moves that where found. Temperature just gave it an opportunity to find another mate in one the same game. Making blunder moves also has an advantage: Obvious MCTS did not realize early enough that the move was a blunder, so many playouts wend into it. By playing the actual move, it will generate a game where the blunder was made, and the network can learn about the blunder position. If the blunder is one move deep, that indeed doens't make much sense, but if the blunder position was 4 moves deep, this is a huge help. Even 1 move deep, the value output can still learn valuable lessons.

Remember: Temperature makes it play different moves, but training still happens on the MCTS output, and it does not affect the move the network learns during training.

Also, temperature might affect the winner of the game. This is a little bad. However, it will also teach the network about positions where it almost cannot make blunders are better than positions where many moves are mistakes. This is actually valuable for the value output.

Ofc it would be interesting to see training with T=1 and T=0.25 and T=0 and compare the difference in strength, but outright dismissing temperature because it feels counter intuitive is not right.

[Akababa]

IIRC the AGZ paper used tau=1 for self-play and tao=0 for evaluation. I agree with @Dorus that tau=1 is fine for training. In fact it may even be optimal because the network needs to train on bad positions just as much as it needs to train on good positions, and tau=1 puts those in the training data in the same ratio as they are encountered in MCTS.

[luigio]

A simple idea for randomness in evaluation games: for every move, choose randomly between running _n_ playouts and running _n_+_m_ playouts, with a small _m_.

[jkiliani]

Sorry but that's a bad idea. If you actually track how often the most visited move changes with increasing playouts, you'll notice that this happens only a few times during the search. If you specify a small m, the chance of this variance in playouts changing the most visited move is minuscule for any given position. That's just not enough randomness for any realistic application.

[luigio]

I guess _m_ could be as big as needed.

[jkiliani]

I just implemented this in #267 , although not yet reviewed and throughly tested.