Keras: Anyone has successful experience of fine-tuning the pre-trained ResNet50

[junfenglx]

Hello:
I follow the blog
https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
to train a small dataset using pre-trained models.
I get a high accuracy when use pre-trained VGG16 model.
However when I use the pre-trained ResNet50 model,
I get a very low accuracy, lower than just train a small conv model from scratch.
I use the output of the last ave pooling layer of ResNet50 as bottleneck features.

Anyone can tell me the reason?
From residual paper I know ResNet50 performs better than VGG16 on Image-Net dataset.
I wonder maybe the pre-trained weight of last dense layer of ResNet50 is very important,
and the extracted features don't have much means.
I also tried to use the last output of conv layer as features, but got same result.
Thanks!

Here is the code:

'''This script goes along the blog post
"Building powerful image classification models using very little data"
from blog.keras.io.
From https://gist.github.com/fchollet/f35fbc80e066a49d65f1688a7e99f069
'''
import os
import h5py
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.utils.visualize_util import plot

from resnet50 import ResNet50

# path to the model weights file.
top_model_weights_path = './model/resnet_bottleneck_fc_model.h5'
# dimensions of our images.
img_width, img_height = 150, 150

train_data_dir = 'data/train'
validation_data_dir = 'data/validation'
nb_train_samples = 4786
nb_validation_samples = 1196
nb_epoch = 50
bottleneck_features_train_path = './model/resnet_bottleneck_features_train.npy'
bottleneck_features_validation_path = './model/resnet_bottleneck_features_validation.npy'


def save_bottlebeck_features():
    datagen = ImageDataGenerator(rescale=1./255)

    # build the ResNet50 network
    model = ResNet50(include_top=False, img_height=img_height, img_width=img_width)
    print('Model loaded.')
    current_dir = os.path.dirname(os.path.realpath(__file__))
    model_path = os.path.join(current_dir, "resnet_50_bottlebeck.png")
    plot(model, to_file=model_path, show_shapes=True)
    # raise ValueError("Stop here")

    generator = datagen.flow_from_directory(
            train_data_dir,
            target_size=(img_width, img_height),
            batch_size=32,
            class_mode=None,
            shuffle=False)
    bottleneck_features_train = model.predict_generator(generator, nb_train_samples)
    np.save(bottleneck_features_train_path, bottleneck_features_train)

    generator = datagen.flow_from_directory(
            validation_data_dir,
            target_size=(img_width, img_height),
            batch_size=32,
            class_mode=None,
            shuffle=False)
    bottleneck_features_validation = model.predict_generator(generator, nb_validation_samples)
    np.save(bottleneck_features_validation_path, bottleneck_features_validation)


def train_top_model():
    datagen = ImageDataGenerator(rescale=1. / 255)
    generator = datagen.flow_from_directory(
        train_data_dir,
        target_size=(img_width, img_height),
        batch_size=32,
        class_mode='categorical',
        shuffle=False)
    labels = []
    i = 0
    for _, y in generator:
        i += len(y)
        labels.append(y)
        # print(i)
        if i == nb_train_samples:
            break
    labels = np.concatenate(labels)
    train_data = np.load(bottleneck_features_train_path)
    train_labels = labels

    generator = datagen.flow_from_directory(
        validation_data_dir,
        target_size=(img_width, img_height),
        batch_size=32,
        class_mode='categorical',
        shuffle=False)
    labels = []
    i = 0
    for _, y in generator:
        i += len(y)
        labels.append(y)
        if i == nb_validation_samples:
            break
    labels = np.concatenate(labels)
    validation_data = np.load(bottleneck_features_validation_path)
    validation_labels = labels
    del labels, generator, datagen, i

    print('train_data.shape: {}, validation_data.shape: {}'.format(train_data.shape, validation_data.shape))
    print('train_labels.shape: {}, validation_labels.shape: {}'.format(train_labels.shape, validation_data.shape))

    model = Sequential()
    model.add(Flatten(input_shape=train_data.shape[1:]))
    # model.add(Dense(256, activation='relu'))
    # model.add(Dropout(0.5))
    model.add(Dense(10, activation='softmax'))

    model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

    model.fit(train_data, train_labels,
              nb_epoch=nb_epoch, batch_size=32,
              validation_data=(validation_data, validation_labels))
    model.save_weights(top_model_weights_path)


save_bottlebeck_features()
train_top_model()

I also tried img_width, img_height = 224, 224, but it doesn't improve accuracy.

Thanks again!

[tglines]

@junfenglx were you able to figure this out?

[junfenglx]

@tglines Maybe it depends the dataset.

[Grimi94]

@junfenglx for how many epochs did you train it? I had a similar issue while fine tuning VGG it took me 2000 epochs to get good results

[oxydron]

Maybe the problem could be 2 things:

1. How much layers did you freeze (ResNet50 have 175 layers to freeze, different than VGG-16)?
2. How different is your problem domain compared to imagenet?

[sebastianbk]

I used Keras' implementation of ResNet50 in this repository and got loss and accuracy statistics comparable to VGG16 on a variety of datasets I have lying around.

I didn't read through your code but if you're still having this issue, maybe try out Keras' implementation?

[oxydron]

https://pastebin.com/dDANxSjz

Here I do transfer learning and fine tuning. The script is very generic and maybe can help you guys.

[izelina]

I came across a similar issue. My base ResNet50 model was frozen, I only wanted to train the dense layer I’ve appended to the model.
The reason my dense layer was not improving was because the values in it were randomly initialized.
The moment I have used a pre-trained dense layer everything started to work.
You mention this great post by Francois Chollet from The Keras Blog: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
There Francois advises that all layers should start with pre-trained weights. Including the dense layer.
I found that this advice worked for me.