Vision: Most pretrained networks have zero accuracy

[wielandbrendel]

The top-1 accuracy of most (but not all) pretrained models is close to zero,

Model            Top1 Acc
-------------  ----------
alexnet              0.52
densenet121          0
densenet161          0
densenet169          0
densenet201          0
resnet18             0
resnet34             0
resnet50             0
resnet101            0.02
resnet152            0
squeezenet1_0        0.62
squeezenet1_1        0.54
vgg11                0.64
vgg11_bn             0.06
vgg13                0.69
vgg13_bn             0.05
vgg16                0.69
vgg16_bn             0.02
vgg19                0.65
vgg19_bn             0.03

This was the output of the following test script (run on Python 3.5 with pytorch 0.2.0.1 and latest master of pytorch vision):

import torch
from torch.autograd import Variable
import torchvision.models as models
from PIL import Image
import numpy as np

models = [models.alexnet,
          models.densenet121,
          models.densenet161,
          models.densenet169,
          models.densenet201,
          models.resnet18,
          models.resnet34,
          models.resnet50,
          models.resnet101,
          models.resnet152,
          models.squeezenet1_0,
          models.squeezenet1_1,
          models.vgg11,
          models.vgg11_bn,
          models.vgg13,
          models.vgg13_bn,
          models.vgg16,
          models.vgg16_bn,
          models.vgg19,
          models.vgg19_bn]

def preprocess_fn(x):
    x = x / 255
    x -= np.array([0.485, 0.456, 0.406])[None,:,None,None]
    x /= np.array([0.229, 0.224, 0.225])[None,:,None,None]
    return x.copy()

import glob
images = glob.glob("images/imagenet/*.png")
data = []

for model in models:
    net = model(pretrained=True).cuda()

    top1 = 0

    for image_path in images[:100]:
        image = Image.open(image_path)
        image_size = 224
        image = image.resize((image_size, image_size), Image.ANTIALIAS)
        image = np.array(image).astype(np.float32)
        image = np.transpose(image, [2,0,1])
        label = int(image_path.split('.')[0].split('_')[1])

        image = torch.from_numpy(preprocess_fn(image[None])).cuda()
        image = Variable(image, volatile=True)
        q = net(image).data.cpu().numpy()[0]

        top1 += np.argmax(q) == label

    data.append([model.__name__, top1/100])
    print('{} reached Top-1 accuracy of {}.'.format(model.__name__, top1 / 100))

[wielandbrendel]

I forgot to put the models into training mode (net.eval() after instantiation of the model). Now the results are all good.