Models: Tensor name "aux_logits/Conv/biases" not found in checkpoint files

[dongb5]

Hi guys!
When I tried to restore the inception_v3 model from the file 'model.ckpt-157585', the error
tensorflow.python.framework.errors.NotFoundError: Tensor name "aux_logits/Conv/biases" not found in checkpoint filesoccurred.

I was running the codes like
saver = tf.train.Saver()
saver.restore(sess, ckpt_file_name)
under the dir 'inception/inception/slim'.

Did I misuse any of them?

[poindextrose]

@dongb5 , I ran into a similar situation when modifying inception_v3. If you have removed the batch normalization parameters from your model, then you will get this error. Did you modify the model?

[michaelisard]

Closing due to lack of activity. Please reopen if there is still a problem.

[flyfj]

encountered the same issue. was fine-tuning inception-v3 using similar code like below:
restore weights:
self.vars_to_restore = slim.get_variables_to_restore( exclude=model_params.custom["restore_scopes_exclude"])
train only the final layers (used when create slim.learning.create_train_op):
self.vars_to_train = [] for scope in model_params.custom["train_scopes"]: variables = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope) self.vars_to_train.extend(variables)
after training, when loading ckpt file, not found error was reported on BatchNorm/beta. Does the above operations actually changed the model? @poindextrose

Figured it out. due to preprocessing, the image size is changed which results in different batchnorm kernel size.

[ImmanuelXIV]

Hi guys, hi @poindextrose! Quick question: I trained inception_v3 from scratch and now when I want to use my generated checkpoint I get the following error (although I did not change batch normalization):

tensorflow.python.framework.errors.NotFoundError: Tensor name "InceptionV3/Conv2d_2a_3x3/BatchNorm/beta" not found in checkpoint files

Is there a way how I can still use my checkpoint? How did you proceed? Thank you

[erebus1]

Same Issue when I do model fine tuning.
@ImmanuelXIV, did you find any solution?

No modification were made to the model.

Here is my question on StackOverflow:
http://stackoverflow.com/questions/40897233/how-continue-train-inception-model-from-checkpoint-in-tensorflow

[tobiaslorenz]

Same here - did anybody find a solution?

[sevazhidkov]

+1

[sevazhidkov]

I fixed it using arg scope:

with slim.arg_scope(slim.variables.all_variables(), device='/cpu:0', batch_norm_params=dict(weight_decay=0.00004, stddev=0.1, batch_norm_decay=0.9997, batch_norm_epsilon=0.001))

But now I'm getting this error:
Tensor name "mixed_17x17x768d/branch7x7dbl/Conv_1/biases" not found in checkpoint files data/inception-v3/model.ckpt-157585

[mnaranjo]

Hi,

I'm getting this error:
Tensor name "mixed_17x17x768c/branch1x1/Conv/BatchNorm/moving_variance/ExponentialMovingAverage" not found in checkpoint files inception-v4/model.ckpt-12027

I use a model fine tuning with 5 classes (flowers) and I'm trying to export the model in tensorflow serving
bazel-bin/tensorflow_serving/example/inception_export --checkpoint_dir=inception-v4 --export_dir=inception-export-v4

when I'm getting the error.

thanks!!

[azagovora]

The same issue when I export the model in tensorflow serving.

[suryaprakaz]

@mnaranjo Any luck? Getting the same error

[mnaranjo]

Hi @suryaprakaz ,
Due to my job, I could not progress with this error. Sorry, but I can not help you.

Good luck with the problem.

[suryaprakaz]

I was able to fix it with inceptionv3 arg_scope

with slim.arg_scope(inception_v3.inception_v3_arg_scope()):
logits, _ = inception_v3.inception_v3(images, NUM_CLASSES+1)

[chanyixialex]

@suryaprakaz why it works?