Models: Problem restoring checkpoint of inception_resnet_v2

[rAm1n]

Hi,

I was just trying to restore the weights from inception_resnet_v2 with the checkpoint provided. It seemed that some of the variables does not exist in the checkpoint and I was wondering if someone could update the checkpoint and make it compatible with the source code.

my Tensorflow version: ('v1.2.0-5-g435cdfc', '1.2.1')

I had a cascade of errors all similar to this:

NotFoundError (see above for traceback): Tensor name "InceptionResnetV2/Conv2d_2b_3x3/biases" not found in checkpoint files inception_resnet_v2_2016_08_30.ckpt
         [[Node: save_1/RestoreV2_42 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_42/tensor_names, save_1/RestoreV2_42/shape_and_slices)]]
         [[Node: save_1/RestoreV2_545/_1427 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3642_save_1/RestoreV2_545", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

so I compared the model with the checkpoint and here are the missing variables:

InceptionResnetV2/AuxLogits/Conv2d_1b_1x1/biases
InceptionResnetV2/AuxLogits/Conv2d_2a_5x5/biases
InceptionResnetV2/Block8/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Block8/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Block8/Branch_1/Conv2d_0c_3x1/biases
InceptionResnetV2/Conv2d_1a_3x3/biases
InceptionResnetV2/Conv2d_2a_3x3/biases
InceptionResnetV2/Conv2d_2b_3x3/biases
InceptionResnetV2/Conv2d_3b_1x1/biases
InceptionResnetV2/Conv2d_4a_3x3/biases
InceptionResnetV2/Conv2d_7b_1x1/biases
InceptionResnetV2/Mixed_5b/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Mixed_5b/Branch_1/Conv2d_0b_5x5/biases
InceptionResnetV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/biases
InceptionResnetV2/Mixed_6a/Branch_0/Conv2d_1a_3x3/biases
InceptionResnetV2/Mixed_6a/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Mixed_6a/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Mixed_6a/Branch_1/Conv2d_1a_3x3/biases
InceptionResnetV2/Mixed_7a/Branch_0/Conv2d_0a_1x1/biases
InceptionResnetV2/Mixed_7a/Branch_0/Conv2d_1a_3x3/biases
InceptionResnetV2/Mixed_7a/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Mixed_7a/Branch_1/Conv2d_1a_3x3/biases
InceptionResnetV2/Mixed_7a/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Mixed_7a/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Mixed_7a/Branch_2/Conv2d_1a_3x3/biases
InceptionResnetV2/Repeat/block35_1/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_1/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_1/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_1/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_1/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_1/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat/block35_10/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_10/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_10/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_10/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_10/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_10/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat/block35_2/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_2/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_2/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_2/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_2/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_2/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat/block35_3/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_3/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_3/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_3/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_3/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_3/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat/block35_4/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_4/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_4/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_4/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_4/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_4/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat/block35_5/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_5/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_5/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_5/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_5/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_5/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat/block35_6/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_6/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_6/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_6/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_6/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_6/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat/block35_7/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_7/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_7/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_7/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_7/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_7/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat/block35_8/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_8/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_8/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_8/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_8/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_8/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat/block35_9/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat/block35_9/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_9/Branch_1/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_9/Branch_2/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat/block35_9/Branch_2/Conv2d_0b_3x3/biases
InceptionResnetV2/Repeat/block35_9/Branch_2/Conv2d_0c_3x3/biases
InceptionResnetV2/Repeat_1/block17_1/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_1/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_1/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_1/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_10/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_10/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_10/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_10/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_11/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_11/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_11/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_11/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_12/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_12/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_12/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_12/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_13/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_13/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_13/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_13/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_14/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_14/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_14/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_14/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_15/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_15/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_15/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_15/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_16/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_16/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_16/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_16/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_17/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_17/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_17/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_17/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_18/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_18/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_18/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_18/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_19/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_19/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_19/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_19/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_2/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_2/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_2/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_2/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_20/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_20/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_20/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_20/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_3/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_3/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_3/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_3/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_4/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_4/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_4/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_4/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_5/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_5/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_5/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_5/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_6/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_6/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_6/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_6/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_7/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_7/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_7/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_7/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_8/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_8/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_8/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_8/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_1/block17_9/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_1/block17_9/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_1/block17_9/Branch_1/Conv2d_0b_1x7/biases
InceptionResnetV2/Repeat_1/block17_9/Branch_1/Conv2d_0c_7x1/biases
InceptionResnetV2/Repeat_2/block8_1/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_2/block8_1/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_2/block8_1/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Repeat_2/block8_1/Branch_1/Conv2d_0c_3x1/biases
InceptionResnetV2/Repeat_2/block8_2/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_2/block8_2/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_2/block8_2/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Repeat_2/block8_2/Branch_1/Conv2d_0c_3x1/biases
InceptionResnetV2/Repeat_2/block8_3/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_2/block8_3/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_2/block8_3/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Repeat_2/block8_3/Branch_1/Conv2d_0c_3x1/biases
InceptionResnetV2/Repeat_2/block8_4/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_2/block8_4/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_2/block8_4/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Repeat_2/block8_4/Branch_1/Conv2d_0c_3x1/biases
InceptionResnetV2/Repeat_2/block8_5/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_2/block8_5/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_2/block8_5/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Repeat_2/block8_5/Branch_1/Conv2d_0c_3x1/biases
InceptionResnetV2/Repeat_2/block8_6/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_2/block8_6/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_2/block8_6/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Repeat_2/block8_6/Branch_1/Conv2d_0c_3x1/biases
InceptionResnetV2/Repeat_2/block8_7/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_2/block8_7/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_2/block8_7/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Repeat_2/block8_7/Branch_1/Conv2d_0c_3x1/biases
InceptionResnetV2/Repeat_2/block8_8/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_2/block8_8/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_2/block8_8/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Repeat_2/block8_8/Branch_1/Conv2d_0c_3x1/biases
InceptionResnetV2/Repeat_2/block8_9/Branch_0/Conv2d_1x1/biases
InceptionResnetV2/Repeat_2/block8_9/Branch_1/Conv2d_0a_1x1/biases
InceptionResnetV2/Repeat_2/block8_9/Branch_1/Conv2d_0b_1x3/biases
InceptionResnetV2/Repeat_2/block8_9/Branch_1/Conv2d_0c_3x1/biases

[rAm1n]

for the future readers:
the problem was that I forgot to use proper arg_scope.
Thanks!

[NoNamesLeft4Me]

Hi, @rAm1n Could you explain more about how you solved this problem? I just encounter exactly the same problem. I am trying to use the following code to load the pre-trained model, and I find out that the model graph generates lots of tensors that named biases, which are not in the pre-trained checkpoint file:

mages, labels = tf.train.batch([image, label], batch_size=batch_size, num_threads=8, capacity=5 * batch_size)

logits, _ = inception_resnet_v2.inception_resnet_v2(images, is_training=False, create_aux_logits=True)
correct_top5 = tf.nn.in_top_k(logits, labels, k=5)

exclude = []
varaibles_to_restore = slim.get_model_variables()

init_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
sess=tf.Session()
sess.run(init_op)

saver = tf.train.Saver(varaibles_to_restore)
test_path = '../test_incepetion_resnet_v2/initial'
saver.save(sess, test_path)

ckp_file_name = '../pre_trained_models/inception_resnet_v2_2016_08_30.ckpt'
saver.restore(sess, ckp_file_name)

It seems that only biases of many layers are missing in the pre-trained model, or they are named differently.

Could you share how you fix this problem with proper arg_scope ?

Thanks

[rAm1n]

Hi @NoNamesLeft4Me

I was just trying the same thing that you're trying to do. When building the model, slims needs to know how to configure the details of each layer. You can do it by providing an arg_scope like inception_arg_scope function.

 with slim.arg_scope(inception_arg_scope()): 
      logits, _ = inception_resnet_v2.inception_resnet_v2(images, is_training=False, create_aux_logits=True)

[NoNamesLeft4Me]

Thanks, @rAm1n . It works smoothly now. I didn't realize the arg_scope would change the actual structure of the network, but it turns out biases tensors are gone and instead batchNorm tensors presents.

[MarkGreeny]

@rAm1n @NoNamesLeft4Me

I was having the same issue you guys had. I than followed your advice. But the only difference is that now we have an error

Tensor name "InceptionResnetV2/Conv2d_1a_3x3/BatchNorm/beta" not found in checkpoint files

Any advice on this?

[TekayaNidham]

@rAm1n @NoNamesLeft4Me

I was having the same issue you guys had. I than followed your advice. But the only difference is that now we have an error

Tensor name "InceptionResnetV2/Conv2d_1a_3x3/BatchNorm/beta" not found in checkpoint files

Any advice on this?

@MarkGreeny did you solve it ?