Keras: What's the difference between BatchNormalization mode=1 and Layer Normalization?

[Kyubyong]

To me, layer normalization (https://arxiv.org/pdf/1607.06450v1.pdf) looks the same as the sample-wise batch normalization in Keras BatchNormalization mode=1. What's their difference?

[MartinThoma]

It seems as if mode was removed in Keras 2.0: https://keras.io/layers/normalization/ ? (By the way: Is there a migration guide?)

[rkoppula]

Yes, some documentation with examples to migrate BatchNormalization layers will be helpful.

[rkoppula]

I found this in the 2.0 release notes - "(use batch metrics for feature-wise normalization during training, and use moving metrics for feature-wise normalization during testing)."

If I am interpreting it right, it is not possible to get the old behavior. Is that right?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[Irtza]

I can't seem to replicate behavior for Batch-normalization Layer mode=2 , in keras 2. Can someone help? @rkoppula , @Kyubyong @MartinThoma .

[HitLuca]

@Irtza

Referencing this solution the suggested way for fixing the new BatchNormalization is to first copy the keras class into another new custom class and then change this

return K.in_train_phase(normed_training,
                                normalize_inference,
                                training=training)

to this

return K.in_train_phase(normed_training,
                                normalize_inference,
                                training=True)

from my own experiments (WGAN-GP on MNIST), the new implementation seems to work. Let me know if it solved your problem. Please be aware that it will only work in "mode 2" now.

[deepitapai]

@HitLuca

Does this mean that I don't need to specify mode=2 while calling this custom BatchNormalization2() ?

[HitLuca]

@deepitapai yes, in this case you don't have to. Please take note that it may not have the behavior you need, it works for GAN training at least. This workaround is needed when using the newer Keras versions, as the mode parameter has been removed.

UPDATE:
Please use the BN implementation in this repository instead as it proved to work perfectly contrary to the slight inaccuracies in the other fix