Fairseq: Accuracy drop after adding quant_noise to InceptionResnetV1

[jmatak]

What is your question?

Since the open-source version of product quantization in the computer vision models is not yet available, after the Quant-Noise paper understanding I have tried following the documentation on implementing both quant noise training and later product quantization on face recognition InceptionResnetV1 pre-trained model.

The main problem is, after applying quant noise on both the Convolutional and Linear layers, the training seems to be broken, a loss is not decreasing as It is in the model version without noise, and accuracy seems to be 0.00%. In addition, a drop in the performance is not visible when a model is just quantized with iPQ. When will example models be available on ClassyVision, and is there a fix to this problem right now?

Code

def quantize_nodes(model):
    conv_blocks = {(3, 3): 9, (1, 1): 1, (1, 7): 7, (7, 1): 7,
                   (1, 3): 3, (3, 1): 3}
    modules = list(model.modules())

    for i in range(len(modules)):
        if (isinstance(modules[i], torch.nn.Conv2d)):
            if (modules[i].kernel_size != (1, 1)):
                modules[i] = quant_noise(modules[i], p=0.1, block_size=conv_blocks[modules[i].kernel_size])
        if (isinstance(modules[i], torch.nn.Linear)):
            modules[i] = quant_noise(modules[i], p=0.1, block_size=8)

resnet = InceptionResnetV1(pretrained='casia-webface).to(device)
quantize_nodes(model=resnet)

What have you tried?

Tried adding noise only to certain convolutional layers (3,3) or linear layers, which reduces the problem but does not fix it.

What's your environment?

• fairseq Version (master):
• PyTorch Version (1.5)
• OS (Ubuntu 18.4):
• How you installed fairseq (git):
• Python version: 3.6

[pierrestock]

Hi jmatak,

Thanks for your interest in our work and thanks for reaching out! Some suggestions below for the Inception, as the parameters we specify in the appendix of our paper are only valid for EfficientNets:

• As a sanity check, could you confirm that the training goes well with p=0 (i.e. no Quant-Noise)?
• In our experience, with ResNets you don't necessarily need Quant-Noise with 3x3 convolutions since the filters are already redundant along the spatial dimension (see previous paper). I would recommend applying Quant-Noise with p=0.05 to only the 1x1 convolutions that are large enough (all except the first layer for instance).
• You could also try slowly increasing p fro 0 to say 0.05 or 0.1 during the training.
• We are working on a new and cleaner API and aim at open sourcing the vision models and code for various tasks as soon as possible.

Let me know if that helps,

Pierre

[jmatak]

Hi @pierrestock,
Thanks for your fast reply. I understand that I am heading upfront in terms of using the model that was not yet tested. I tried to replicate to all your questions by implementing and trying in practice:

• With Quant-Noise parameter p=0, training goes well, accuracy increases to 99
• Well, in the Inception architecture, there is an error in adding Quant-Noise to the 1x1 convolutions. The problem is in the calculated input size per channel. This kind of error:
  Calculated padded input size per channel: (38 x 38). Kernel size: (80 x 64). Kernel size can't be greater than actual input size

in this original code part:

if mod.kernel_size == (1, 1):
           mask = torch.zeros(int(in_channels // block_size * out_channels), device=weight.device)
           mask.bernoulli_(p)
           mask = mask.repeat_interleave(block_size, -1).view(-1, in_channels)

• Since the majority of convolutions in Inception are 3x3, removing them, Linear layers, and 1x1 convolutions because of the bug, the training process starts working well, but it only affects a minority of 1x7 and 1x3 convolutions. Basically, the code is still broken. For more guidance in this problem, I tried to train with only adding Quant-Noise to Linear layers, but no progress is made.

• I hope the API will be available as soon as possible, looking forward to this work!

[jmatak]

Anyone, when will new API be publically available?