Stockfish: Discussion on asymmetry (issues, ideas, testing plans)

[NKONSTANTAKIS]

An important, controversial approach to chess is an asymmetric one, unique to each side: W and B.

Such an approach de-facto means introducing bias, as the same position will be regarded differently from each side. This bias will be primarily derived by openings, (through maximizing performance individually) or lack of them (as starting position inherently induces bias through the dynamic asymmetry of a tempo). This means that while for a random position a symmetric approach is adequate and correct, for the starting position its neglecting to take into account the tempo difference.

Theoretically chess is solvable with both approaches, but with practical limitations and incomplete information, various strategies are used in an effort to approximate perfection.

From a chess perspective the asymmetric approach enables color-exclusive optimization & strategy making for maximizing results. This is especially true for the field of use, where given openings and positions are not randomly generated, but very highly correlate with the starting position.

From a coding perspective the symmetric approach offers easier handling of code, a simpler unified eval output, and other human conveniences, so its the commonly used method. Neural networks on the other hand, feature automated machine learning, hence convenience is not needed, and asymmetry is included naturally.

This thread is dedicated to promoting implementation of asymmetric approach, by overcoming the technical difficulties and devising ways for unlocking the potential.

[NKONSTANTAKIS]

Continuation from discussion following an asymmetric pull request. https://github.com/official-stockfish/Stockfish/pull/2608#issuecomment-607369620

[danegraphics]

Asymmetry is a result of the tempo differences at the beginning of the game resulting in different average structures between white and black. It is these structures that change the advantages and disadvantages of different positions, and not simply whether it is white or black that is playing.

This means that thankfully, the huge disadvantages of programming for asymmetric eval can be completely avoided while still gaining the advantages by simply changing the eval based on structural/positional circumstances, instead of simply based on whether or not it's black or white.

Because of the symmetric nature of chess, any asymmetric eval will inevitably introduce flaws. It may assist in the _average_ position, but it will almost always hurt in non-average positions.

TLDR: Asymmetric eval is a bad idea because it introduces flaws and can _always_ be implemented better in a symmetric form.

[xoto10]

But the "better" implementation may be nicer to read, but slower to run in practice, giving worse performance in real life.

[NKONSTANTAKIS]

@danegraphics You say that the asymmetric approach will hurt non-average positions. How is this possible? The split evolution will reach to the same conclusions, when applicable. If you are talking about analysing a position, the best tool would probably be the exact middle of the 2 sides and not the color specific one.

This concept of middle point can be the solution to most concerns: Simultaneously with development of the 2 sides, the middle point will always be available, easily produced, simple as ever, and continuing the progress of symmetric SF as a byproduct!

As both sides happen 50% , the exact sum of every asymmetric optimization divided by 2 will always closely match the symmetric evolution. But with the benefit of much easier and distinct patch selection. In a way very similar to what we are currently doing with contempt: evolving C=24 which is easier and trusting that C=0 will also evolve.

[StevePalm01]

Symmetry is a static observation and normally it is just partial in a position and has all but the same dynamic effect for the 2 sides. Especially Kf1/Kf8 for White and Black respectively discussed/tested a while back is totally different: for White, this normally (apart from some forced KGA variations) e.g. in French involves a very active plan of pushing the h-pawn and preparing for a rook lift (that may not be necessary, but it is a powerful option). One can see this frequently in NN/Leela games, but also in human games (see e.g. MVL in the recent aborted Candidates tournament). Whereas for Black, moving the King to Kf8 is purely defensive and may effectively lead to trapping the rook and making it totally useless.

In view of the above, I would encourage i.e. give bonus for White for a h4 push and rook lift plan (possibly involving Kf1), but give significant malus for Black for Kf8 (unless it is in a line that involves h5 and rook lift). SF sometimes goes for active plans as White, but encouraging this behavior further would in my opinion give SF more chances to play dynamic positions where it generally excels

[NKONSTANTAKIS]

As the majority of the devs seems to be skeptical/unwilling to trot on this path on their own, and due to me being a chess guy and not a coder, I will try opening some channels of communication/cooperation and organization. I draw this motivation both from my strong belief that the asymmetric way is superior and from @vondele 's encouragement and interest on it.

Some ways to start rolling, presented in order of degree of community's commitment:

1. Create an asymmetric branch and allocate some resources in it. Update it with master when little-no conflict (search patches are 100% mutual!). Occasionally check both its own progress vs master, and its symmetric byproduct's progress (midpoint of W-B deviation). This way any progress of the asymmetric branch could be transferable to the main one. For example I think its a logical estimation that for every 5 asymmetric elo gained, the symmetric midpoint will gain 2 elo.

2. Make a serious high elo attempt with a big asymmetric tune. PSQT's make sense, probably all at once, or the most important ones (pawn & king & maybe rook). This, if done right, could give an idea of the potential, but if its a tame attempt its a high chance that once more the elo gain will not be enough to counter the pain of surrendering symmetry. That's why I think "GO BIG OR GO HOME" applies here.

3. Gather information in an academic way for promoting understanding of the situation. Results could be interesting and useful for selecting superior configurations at tournaments and matches. The target would be parameters which are more likely to differ for B and W.

For example contempt besides gaining elo vs weaker and losing vs stronger, supposedly helps when ahead and backfires when behind. So, as white is typically ahead and black behind, a simple idea would be to try higher with white and lower with black. So we could check the degree of power of this concept, by testing asymmetric C24/C0 vs C24, and only if the result is strong continue vs C12(midpoint) and vs C0. This can be done cheaply at fixed 60K games 10" or 20" .

1. Work locally, trying to produce a version clearly superior to master. This is for motivated people who are either apt at sophisticated tuning (@lantonov etc.) or with a keen intuition for hand-picking values (@FauziAkram etc.), or anyone giving it a shot by generalizing fail strategic trends of SF, commonly highlighted vs NN's. I think of it as tutti-frutti combos of slight logical adjustments, hoping to hit the jackpot. Those which do well locally (big error margins), can be tried on fishtest. Or even straight at fishtest, as filtering is the use of STC.

2. Do nothing and just wait until an asymmetric patch becomes lucky/good enough to pass STC+2xLTC , premiering and thus enabling normal STC+LTC for subsequent ones.

I welcome everyone's opinions for each of the 5 methods separately, suggestions for improving them, and also expressing willingness to invest on each, and speculating chance of success & worth of investment of resources.

[Alayan-stk-2]

For example contempt besides gaining elo vs weaker and losing vs stronger, supposedly helps when ahead and backfires when behind. So, as white is typically ahead and black behind, a simple idea would be to try higher with white and lower with black. So we could check the degree of power of this concept, by testing asymmetric C24/C0 vs C24, and only if the result is strong continue vs C12(midpoint) and vs C0. This can be done cheaply at fixed 60K games 10" or 20" .

SF has already has dynamic contempt that increases contempt when ahead and lowers it when behind, that is shown to be elo-gaining independently of the base contempt.

[noobpwnftw]

The only problem is that there seems be no end to the argument, and people shouldn't be launching contempt tests every other week to see if it still works. There are a lot of holes to plug, don't get attached to one thing and one thing only may be more productive.

[noobpwnftw]

As far as I understand, such asymmetric behavior can be described by tempo, sometimes it is worth more and sometimes worth less, while it does not naturally occur for each side equally often.
Dynamic tempo, maybe?

[NKONSTANTAKIS]

@Alayan-stk-2 Yes I am aware of that, but why would eval be the only parameter determining the dynamic contempt formula? White and black might benefit from an exclusive formula or base contempt. But this is a minor issue, contempt test was just a possible example and not main focus.

@noobpwnftw The dynamic effect of 1 tempo leads to average structural differences, SF is particularly weak at openings due to high branching factor and less tactics. A customized, asymmetric eval would benefit most of all the opening, losing value as game progresses.

[StevePalm01]

Mobility (including tempi) vs. material is interesting asymmetry. E.g. the current Leela net seems to go for pawn sacs, say move pawn to g4 to open up the file creating long-term pressure. Such behavior i.e. sacrificing material for initiatve was extreme with A0 (I wonder whether a short time control in training favored such approach making it difficult to defend with little time). SF of course understands such lines, but these may not be possible to calculate exactly at the beginning and are thus abandoned in favor of more ”solid” lines. However, giving more weight to mobility may create more opportunities also for SF. Again, aggressive approach may work better with white (most of the wins between the top engines in balanced openings are with white I guess), so here a higher contempt with white among other asymmetric settings may work to the benefit of SF in the opening/middle-game.

Overall, the framework proposed by NKONSTANTAKIS
above seems a systematic way to analyze the opportunities further.

[syzygy1]

The idea proposed here is fundamentally incompatible with alpha-beta.

[NKONSTANTAKIS]

Its just different eval for each side, just like contempt. There had been also worries about TT pollution in analysis, due to eval inconsistency, but it works.

Anyway, due to the complexity of this path, the ugliness of evaluating same position differently upon color, the difficulties of development, the lack of interest, and most importantly the rise of NNUE, which should be a lot more rewarding than this taking priority of using fishtest resources, largely obsolete this.

[syzygy1]

No, contempt doesn't change that white's and black's evaluations of the same position are the same (with only the sign changed).

[NKONSTANTAKIS]

@syzygy1 https://github.com/official-stockfish/Stockfish/pull/2803 Is definitely compatible with alpha-beta, and gaining hard elo too. If your statement stands, it means that it can't be asymmetric. Could you please have a look?

[syzygy1]

Why would you think #2803 is asymmetric??

[syzygy1]

From the discussion of #2803 I now understand what you mean.

If by "asymmetry" you simply mean that, for example, a white knight on A1 is evaluated differently from a black knight on A8, then I agree there is no contradiction with alpha-beta and it would in fact be no problem in SF to deal with such asymmetry.

In this case, both sides still evaluate the same position in the same way. This is anyway inherent in alpha-beta (or minimax for that matter): if white's best move leads to +1.23, this is backed up as -1.23 to the parent node where black is to move. There can be no asymmetry here or the search would fall apart completely.

Edit: I guess I should have checked #2608 before my initial comment that asymmetry is incompatible with alpha-beta. The asymmetry you are referring to here is indeed not problematic at all from the point of view of the search. I apologise for responding negatively without first properly checking what the discussion was really about.