Models: Object Detection Training not reading in batch norm parameters

Created on 28 Jun 2017 · 2Comments · Source: tensorflow/models

I was following the directions to train a ssd_mobilenet model that trains the detection network from scratch after initializing the backbone with the weights of a model trained on Imagenet. I downloaded the pre-trained mobilenet checkpoint for this experiment. However, I get warning messages complaining that the batch norm parameters could not be loaded (see below for the messages). Has anyone else seen this issue or any pointers on how I can try to debug it?

Below I show my train_config to set up the experiment from the stock one given in the detection library and the warning messages I see about not reading in

train_config: {
batch_size: 24
optimizer {
rms_prop_optimizer: {
learning_rate: {
exponential_decay_learning_rate {
initial_learning_rate: 0.001
decay_steps: 800720
decay_factor: 0.95
}
}
momentum_optimizer_value: 0.9
decay: 0.9
epsilon: 1.0
}
}
fine_tune_checkpoint: "/raid/gavenkatesh/Networks/tensorflowModels/models/object_detection/modelImagenet/mobilenet_v1_1.0_224.ckpt"
from_detection_checkpoint: false
data_augmentation_options {
random_horizontal_flip {
}
}
data_augmentation_options {
ssd_random_crop {
}
}
}

Warning messages:
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_2_1x1_256/BatchNorm/beta] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_2_1x1_256/BatchNorm/gamma] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_2_1x1_256/BatchNorm/moving_mean] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_2_1x1_256/BatchNorm/moving_variance] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_2_1x1_256/weights] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_3_1x1_128/BatchNorm/beta] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_3_1x1_128/BatchNorm/gamma] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_3_1x1_128/BatchNorm/moving_mean] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_3_1x1_128/BatchNorm/moving_variance] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_3_1x1_128/weights] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_4_1x1_128/BatchNorm/beta] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_4_1x1_128/BatchNorm/gamma] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_4_1x1_128/BatchNorm/moving_mean] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_4_1x1_128/BatchNorm/moving_variance] not available in checkpoint
WARNING:root:Variable [MobilenetV1/Conv2d_13_pointwise_1_Conv2d_4_1x1_128/weights] not available in checkpoint

System information

What is the top-level directory of the model you are using: object_detection
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
TensorFlow installed from (source or binary): Source
TensorFlow version (use command below):1.2
Bazel version (if compiling from source): 0.5.1
CUDA/cuDNN version: CUDA 8.0, CUDNN 6.0
GPU model and memory: GP100 with 16 GB memory
Exact command to reproduce: I am using the vanilla train.py command from the object detection README: python object_detection/train.py --logtostderr --pipeline_config_path=object_detection/samples/configs/ssd_mobilenet_v1_voc07.config --train_dir=object_detection/modelCheckpoints/pascal/

Describe the problem

Source code / logs

Source

livenletdie

Most helpful comment

Hi @livenletdie - it looks like you are initializing from an Imagenet pretrained classification checkpoint?

If so, don't worry about this; those layers are additional layers added to the Mobilenet backbone by SSD and don't exist in the classification checkpoint.

jch1 on 28 Jun 2017

👍6

All 2 comments

Hi @livenletdie - it looks like you are initializing from an Imagenet pretrained classification checkpoint?

If so, don't worry about this; those layers are additional layers added to the Mobilenet backbone by SSD and don't exist in the classification checkpoint.

jch1 on 28 Jun 2017

👍6

Yes I am initializing from the imagenet retrained classification checkpoint. Thanks for the clarification. That is very helpful.

livenletdie on 28 Jun 2017

Was this page helpful?

0 / 5 - 0 ratings