Icevision: Why torchvision uses FrozenBatchNorm?

[lgvaz]

What is FrozenBatchNorm?

Why is it used on models like FasterRCNN and MaskRCNN?

How does it impact fine tuning? Because it's often a good idea to never freeze any batch norm layer while training (even if the other layers are freezed)

[ai-fast-track]

The short answer is: by unfreezing batchnorm our model get a better accuracy.

Now the why (Example: Transfer Learning using ImageNet):
When we use a pretrained model, the batchnorm layer contains the mean, the standard deviation, the gamma and beta (2 trainable parameters) of the pretrained dataset (ImageNet in the case of images).

If we freeze our batchnorm layer with our dataset, we are feeding our model with our data (our images) and normalizing our batch with ImageNet mean, standard deviation, gamma and beta: Those values are off specially if our images are different from the ImageNet images. Therefore, your normalized activations are also off which leads to less than optimal results.

We keep the batchnorm layer unfrozen because while we are training the model and for each batch we will calculate the mean, the standard deviation of the activations of our data (batch of images), and updating (training) the corresponding gamma and beta, and using those results to normalize our activations of the current batch: The normalized activations are therefore more aligned with our images (dataset) as opposed to those obtained with a frozen batchnorm.

[lgvaz]

by unfreezing batchnorm our model get a better accuracy.

Exactly!! This is why it bugs me out why those models use FrozenBatchNorm!

[ai-fast-track]

Here is a follow-up answer that I also gave at the fastai forum:

I’m quoting Jeremy from lesson 12 about batchnorm:
“Anytime something weird happens to your neural net it’s almost certain it’s because of the batchnorm because batchnorm makes everything weird!”

To answer your question, you might check out the 11a_transfer_learning.ipynb 1 from lesson 12 - Part-2 2019 course. You can also jump to lesson 12 video 1 portion where Jeremy explains the effect of the mean, the standard deviation, and the batchnorm trainable parameters on training a custom model that I am referring to in my previous post.

Here a little summary about the experiment that he showed in that video:
1 - He created a custom head for his model
2 - He froze the whole body (including the batchnorm layers) of the pretrained model.
3 - He trained his model for 3 epochs, and he got 54% accuracy
4 - He unfroze the whole body, and train the model for 5 epochs, and he got 56% accuracy (which was surprising low)

Then, he decided to unfreeze batchnorm from the beginning (meaning while training the custom head). He showed the following steps:
1 - He froze the whole body except the batchnorm layers of the pretrained model.
2 - He trained his model for 3 epochs, and he got 58% accuracy (which is already better than above)
3 - But more importantly, when he unfroze the whole body, and train the model for 5 epochs, he got 70% accuracy (and that’s a huge jump)