Leela-chess: Weird weights in input convolution

[Ttl]

It seems that the batch norm after the input convolution has very big variances. Minimum variance in the latest weights is 86 and maximum is 1.2e6. For comparison in LZ Go BN variances after input convolution are from 0.01 to 1.4. Much more sane values considering that BN will try to set the output variance to 1. If there are really that big values during training or inference it might cause issues with the floating point dynamic range.

Plotting the summed absolute values of each 3x3 filter in input convolution gives following plot:

And sum across the output channels:

Magnitude of weights for second to last input plane (move_count) are huge compared to everything else. Program is probably going to give completely different evaluations whether the commit https://github.com/glinscott/leela-chess/commit/fcd67bd5a4d34c4fceaa85ccf2264c33cf235ae0 is applied or not.

[killerducky]

Weird, chunkparser.py forces move_count=0, effectively ignoring if the client as commit https://github.com/glinscott/leela-chess/commit/fcd67bd5a4d34c4fceaa85ccf2264c33cf235ae0 or not. I assumed this means the move_count plane will be regarded as don't care by the backprop process, and the regularization term on the weights would force them to zero.

As I was typing this I remembered vaguely that BN layers do not have regularization applied? Is that right?

[Ttl]

There's nothing to regularize in BN layers right now since center and scale are disabled. Moving averages and variances are calculated from each batch during the training and updated with momentum of 0.99. If the variance in the weights is 1e6 then the variance was that much on average in each batch during the training.

If the move_count plane was always zero then the weights should go to zero because of the L2 regularization but this doesn't seem to be the case.

[killerducky]

Maybe this is the problem. March 18 the clients started dumping V2 data. March 21 the clients disabled move_count. chunkparser.py seems to only force move_count=0 for V1 data.

commit fcd67bd5a4d34c4fceaa85ccf2264c33cf235ae0
Author: Error323 <[email protected]>
Date:   Wed Mar 21 07:54:36 2018 +0100

    Disable move_count as in training


commit d080cf0285ca6d260feee65f37b48e69ab75b7e4
Author: F. Huizinga <[email protected]>
Date:   Sun Mar 18 20:34:35 2018 +0100

    Training chunk V2 (#119)

    Passes integrity tests for both *nix and Windows.

chunkparser.py conver_v2_to_tuple()

        planes = planes.tobytes() + self.flat_planes[us_ooo] + self.flat_planes[us_oo] + self.flat_planes[them_ooo] + self.flat_planes[them_oo] + self.flat_planes[stm] + self.flat_planes[rule50_count] + self.flat_planes[move_count] + self.flat_planes[0]

[Error323]

Oh no... Thank you @Ttl! And @killerducky, this is indeed a bug on my end. It doesn't set V2 move_count to 0.

I'll fix it right now for training...

[Error323]

I fixed it for training. I'll commit and push in the morning. Thanks again!

[Error323]

Could you revisualize the inputfilters @Ttl? Given this quick fix?

[Ttl]

f5c40:

Variance in input convolution BN is now much smaller. Maximum is 1968.

And the same for supervised weights for comparison:

You can find the script here: https://gist.github.com/Ttl/c10eacc0342603dd02cfa93afdf065ce

I'm not sure if disabling move_count was a good change since it seems that the network used it before. Now the supervised network and the old networks probably don't work correctly anymore with the latest program.

In my opinion the move_count and rule50 planes should have been normalized to same range as the other planes, but it might be too late to change it now. Dynamic range after the input convolution can get pretty large when they are unscaled as seen from the variances. I wonder if it's related to OpenCL errors some people are seeing.

[Error323]

Thanks a lot @Ttl!

I'm not sure if disabling move_count was a good change since it seems that the network used it before. Now the supervised network and the old networks probably don't work correctly anymore with the latest program.

Yes this is unfortunate, though I can retrain the kbb net easily. And it might occur more often as we try to improve the neural network architecture.

In my opinion the move_count and rule50 planes should have been normalized to same range as the other planes, but it might be too late to change it now. Dynamic range after the input convolution can get pretty large when they are unscaled as seen from the variances. I wonder if it's related to OpenCL errors some people are seeing.

It would have been better maybe. The problem was that the move_count has always been clamped to an unsigned byte with a max value of 255, while our max plycount is 450. This is the primary reason that made me remove/disable it (albeit incompletely at first :disappointed:).

The OpenCL errors were there since V1 already. I think they're mostly related to our policy and value head FC layers being computed on the GPU also. But it's very probable that the issue described here worsened the error rate.

Again thank you very much for your analysis :1st_place_medal: