Models: Value Error: First Step Cannot Be Zero

Created on 28 Mar 2018 · 27Comments · Source: tensorflow/models

What is the top-level directory of the model you are using: Object_detection
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 7
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): 1.6.0
Bazel version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce: python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/pipeline.config

When running the faster_rcnn_resnet101_kitti_2018_01_28 model it is throwing the error:

(base) C:\Users\MAT\models-master\research\object_detection>python train.py --lo
gtostderr --train_dir=training/ --pipeline_config_path=training/pipeline.config
C:\Users\MAT\Anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Con
version of the second argument of issubdtype from `float` to `np.floating` is de
precated. In future, it will be treated as `np.float64 == np.dtype(float).type`.

from ._conv import register_converters as _register_converters
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\Users\MAT\models-master\research\object_detection\tra
iner.py:228: create_global_step (from tensorflow.contrib.framework.python.ops.va
riables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\Users\MAT\models-master\research\object_detection\cor
e\box_predictor.py:396: calling reduce_mean (from tensorflow.python.ops.math_ops
) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From C:\Users\MAT\models-master\research\object_detection\cor
e\losses.py:316: softmax_cross_entropy_with_logits (from tensorflow.python.ops.n
n_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

Traceback (most recent call last):
File "train.py", line 167, in <module>
tf.app.run()
File "C:\Users\MAT\Anaconda3\lib\site-packages\tensorflow\python\platform\app.
py", line 126, in run
 _sys.exit(main(argv))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "C:\Users\MAT\models-master\research\object_detection\trainer.py", line 2
55, in train
train_config.optimizer)
File "C:\Users\MAT\models-master\research\object_detection\builders\optimizer_
builder.py", line 50, in build
learning_rate = _create_learning_rate(config.learning_rate)
File "C:\Users\MAT\models-master\research\object_detection\builders\optimizer_
builder.py", line 109, in _create_learning_rate
learning_rate_sequence, config.warmup)
File "C:\Users\MAT\models-master\research\object_detection\utils\learning_sche
dules.py", line 156, in manual_stepping
raise ValueError('First step cannot be zero.')

Source

ashiqks

👍5

Most helpful comment

We recently updated the code so this block

          schedule {
            step: 0
            learning_rate: .0001
          }

is not recognized anymore.

The model configs in model zoo still have this block so it crashed the program. You can remove this block or use the latest config instead of the downloaded one in tar.gz file.

pkulzc on 28 Mar 2018

👍57 🎉10

All 27 comments

We recently updated the code so this block

          schedule {
            step: 0
            learning_rate: .0001
          }

is not recognized anymore.

The model configs in model zoo still have this block so it crashed the program. You can remove this block or use the latest config instead of the downloaded one in tar.gz file.

pkulzc on 28 Mar 2018

👍57 🎉10

@pkulzc removing the block is no good , it still throws no variables to save error .
"ValueError: No variables to save"

vishalgolcha on 29 Mar 2018

@vishalgolcha could you share more details?

pkulzc on 29 Mar 2018

What is the top-level directory of the model you are using: Object_detection
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 17.04
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): 1.6.0
CUDA/cuDNN version *: cuda 9.0
*GPU model and memory: 1080 8GB

i am trying to finetune a faster rcnn inception model for a custom tfrecord file ,
below is my config file
model { faster_rcnn { num_classes: 8 image_resizer { keep_aspect_ratio_resizer { min_dimension: 600 max_dimension: 1024 } } feature_extractor { type: "faster_rcnn_inception_v2" first_stage_features_stride: 16 } first_stage_anchor_generator { grid_anchor_generator { height_stride: 16 width_stride: 16 scales: 0.25 scales: 0.5 scales: 1.0 scales: 2.0 aspect_ratios: 0.5 aspect_ratios: 1.0 aspect_ratios: 2.0 } } first_stage_box_predictor_conv_hyperparams { op: CONV regularizer { l2_regularizer { weight: 0.0 } } initializer { truncated_normal_initializer { stddev: 0.00999999977648 } } } first_stage_nms_score_threshold: 0.0 first_stage_nms_iou_threshold: 0.699999988079 first_stage_max_proposals: 300 first_stage_localization_loss_weight: 2.0 first_stage_objectness_loss_weight: 1.0 initial_crop_size: 14 maxpool_kernel_size: 2 maxpool_stride: 2 second_stage_box_predictor { mask_rcnn_box_predictor { fc_hyperparams { op: FC regularizer { l2_regularizer { weight: 0.0 } } initializer { variance_scaling_initializer { factor: 1.0 uniform: true mode: FAN_AVG } } } use_dropout: false dropout_keep_probability: 1.0 } } second_stage_post_processing { batch_non_max_suppression { score_threshold: 0.0 iou_threshold: 0.600000023842 max_detections_per_class: 100 max_total_detections: 300 } score_converter: SOFTMAX } second_stage_localization_loss_weight: 2.0 second_stage_classification_loss_weight: 1.0 } } train_config { batch_size: 1 data_augmentation_options { random_horizontal_flip { } } optimizer { momentum_optimizer { learning_rate { manual_step_learning_rate { initial_learning_rate: 0.000199999994948 schedule { step: 900000 learning_rate: 1.99999994948e-05 } schedule { step: 1200000 learning_rate: 1.99999999495e-06 } } } momentum_optimizer_value: 0.899999976158 } use_moving_average: false } gradient_clipping_by_norm: 10.0 fine_tune_checkpoint: "/home/lashiv/obj-detect/sop/models/model.ckpt.index" from_detection_checkpoint: true num_steps: 200000 } train_input_reader { label_map_path: "/home/lashiv/obj-detect/sop/data/StringIntLabelMap.pbtxt" tf_record_input_reader { input_path: "/home/lashiv/obj-detect/sop/data/train.record" } } eval_config { num_examples: 8000 max_evals: 10 use_moving_averages: false } eval_input_reader { label_map_path: "/home/lashiv/obj-detect/sop/data/StringIntLabelMap.pbtxt" shuffle: false num_readers: 1 tf_record_input_reader { input_path: "/home/lashiv/obj-detect/sop/data/train.record" } }

the traceback it shows is
Traceback (most recent call last):
File "object_detection/train.py", line 167, in
tf.app.run()
File "/home/lashiv/anaconda3/envs/detect/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "object_detection/train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/home/lashiv/obj-detect/models/research/object_detection/trainer.py", line 285, in train
init_saver = tf.train.Saver(available_var_map)
File "/home/lashiv/anaconda3/envs/detect/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1293, in __init__
self.build()
File "/home/lashiv/anaconda3/envs/detect/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1302, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/lashiv/anaconda3/envs/detect/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1327, in _build
raise ValueError("No variables to save")
ValueError: No variables to save

vishalgolcha on 29 Mar 2018

You're using the wrong file for checkpoint. Use this:
fine_tune_checkpoint: "/home/lashiv/obj-detect/sop/models/model.ckpt"
instead.
Since this is a different problem, please open another thread for this if you have further questions.

pkulzc on 29 Mar 2018

      schedule {
        step: 0
        learning_rate: .0001
      }

i have the same error . and i can find where is the block.could you please tell me ?

chenyuyi94 on 5 Apr 2018

It's in the config file, train_config/optimizer/learning_rate.

pkulzc on 5 Apr 2018

thank you for your reply

在 2018年4月5日，下午11:48，pkulzc notifications@github.com 写道：

It's in the config file, train_config/optimizer/learning_rate.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

chenyuyi94 on 6 Apr 2018

Mr. pkulzc pls be more specific :

The name of file that must be modifyed
The location of this file
if any link avalible for correct model zoo
Best Regards

krassynesh on 5 May 2018

An simpler way is to always use the config here instead of the ones in downloaded file.

For example, If you want to train a faster_rcnn_resnet101_kitti model, use models/research/object_detection/samples/configs/faster_rcnn_resnet101_kitti.config instead of the one in the downloaded compressed file. The configs in git repo are guaranteed to be latest.

pkulzc on 10 May 2018

❤7 👍3 🎉1

Thank you the issue resolved but coming up with new error

(tensorflow1) C:\tensorflow1\models\research\object_detection>python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 93, in main
FLAGS.pipeline_config_path)
File "C:\tensorflow1\models\research\object_detection\utils\config_util.py", line 94, in get_configs_from_pipeline_file
text_format.Merge(proto_str, pipeline_config)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 533, in Merge
descriptor_pool=descriptor_pool)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 587, in MergeLines
return parser.MergeLines(lines, message)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 620, in MergeLines
self._ParseOrMerge(lines, message)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 635, in _ParseOrMerge
self._MergeField(tokenizer, message)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 735, in _MergeField
merger(tokenizer, message, field)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 822, in _MergeMessageField
raise tokenizer.ParseErrorPreviousToken('Expected "%s".' % (end_token,))
google.protobuf.text_format.ParseError: 140:16 : Expected "}"

Shiv1799 on 29 May 2018

@Shiv1799 apparently the config file is missing a "}" , you will want to check it.

pkulzc on 29 May 2018

👍1

hi @pkulzc
could you help me with this?
i have a pretrained model (RCNN with Inception v2 model trained on the COCO dataset) with these files :
checkpoint model.ckpt.data-00000-of-00001
example.config model.ckpt.index
frozen_inference_graph.pb model.ckpt.meta
labels.txt saved_model/

what file should i set to the "fine_tune_checkpoint:" value in my file example.config ?

currently my example.config looks like this :

fine_tune_checkpoint: "/home/antoine/Documents/ia/HereIsWally/trained_model/checkpoint"
it is right ?
I have some mistakes in the training of the model and I suspect that it could be by this checkpoint

antoine29 on 12 Jun 2018

@antoine29 you should use "...trained_model/model.ckpt".
BTW, This is a stackoverflow question since it's neither a bug or a feature request. Please ask there if you have future questions like this.

I'm closing this bug since it's straightforward and has been answered.

pkulzc on 12 Jun 2018

(tensorflow1) C:\tensorflow1\models\research\object_detection>python train.py --
logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_in
ception_v2_pets.config
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\trainer.
py:260: create_global_step (from tensorflow.contrib.framework.python.ops.variabl
es) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 0 to match input file shards.

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 274, i
n train
train_config.prefetch_queue_capacity, data_augmentation_options)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 59, in
create_input_queue
tensor_dict = create_tensor_dict_fn()
File "train.py", line 121, in get_next
dataset_builder.build(config)).get_next()
File "C:\tensorflow1\models\research\object_detection\builders\dataset_builder
.py", line 186, in build
process_fn, config.input_path[:], input_reader_config)
File "C:\tensorflow1\models\research\object_detection\utils\dataset_util.py",
line 141, in read_dataset
sloppy=config.shuffle))
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 973, in apply
dataset = transformation_func(self)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\c
ontrib\data\python\ops\interleave_ops.py", line 76, in _apply_fn
buffer_output_elements, prefetch_input_elements)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 130, in __init__
cycle_length, block_length)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1959, in __init__
super(InterleaveDataset, self).__init__(input_dataset, map_func)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1928, in __init__
self._map_func.add_to_graph(ops.get_default_graph())
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 484, in add_to_graph
self._create_definition_if_needed()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 319, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 336, in _create_definition_if_needed_impl
outputs = self._func(*inputs)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1916, in tf_map_func
dataset = map_func(nested_args)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 196, in __init__
filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1014, in convert_to_tensor
as_ref=False)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 947, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float
32: 'Tensor("arg0:0", shape=(), dtype=float32)'

Getting this error anyone who can help with this??

osama191 on 20 Jun 2018

👍2 👎1

After resolving the first step cannot be zero, I encountered this problem below.

(tensorflow) C:\Users\User\Documents\GitHub\models-master\research\object_detection>python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\h5py__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
WARNING:tensorflow:From C:\Users\User\Documents\GitHub\models-master\research\object_detection\trainer.py:260: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
WARNING:tensorflow:From C:\Users\User\Documents\GitHub\models-master\research\object_detection\core\box_predictor.py:407: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\Users\User\Documents\GitHub\models-master\research\object_detection\core\losses.py:317: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

Traceback (most recent call last):
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 510, in _apply_op_helper
preferred_dtype=default_dtype)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\constant_op.py", line 235, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\constant_op.py", line 214, in constant
value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 441, in make_tensor_proto
_GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 524, in _apply_op_helper
values, as_ref=input_arg.is_ref).dtype.name
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\constant_op.py", line 235, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\constant_op.py", line 214, in constant
value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 441, in make_tensor_proto
_GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\User\Documents\GitHub\models-master\research\object_detection\trainer.py", line 298, in train
train_config.optimizer)
File "C:\Users\User\Documents\GitHub\models-master\research\object_detection\builders\optimizer_builder.py", line 50, in build
learning_rate = _create_learning_rate(config.learning_rate)
File "C:\Users\User\Documents\GitHub\models-master\research\object_detection\builders\optimizer_builder.py", line 109, in _create_learning_rate
learning_rate_sequence, config.warmup)
File "C:\Users\User\Documents\GitHub\models-master\research\object_detection\utils\learning_schedules.py", line 169, in manual_stepping
[0] * num_boundaries))
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\array_ops.py", line 2681, in where
return gen_math_ops.select(condition=condition, x=x, y=y, name=name)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 7189, in select
"Select", condition=condition, t=x, e=y, name=name)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 528, in _apply_op_helper
(input_name, err))
ValueError: Tried to convert 't' to a tensor and failed. Error: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

Any one familiar with this?

RolandOne on 23 Jun 2018

@osama191 i have the same problem. Do you fix it ?

Help me please

WARNING:tensorflow:From /home/luiz/model_ssd/research/object_detection/trainer.py:262: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 0 to match input file shards.
Traceback (most recent call last):
File "train.py", line 188, in
tf.app.run()
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 184, in main
graph_hook_fn=graph_rewriter_fn)
File "/home/luiz/model_ssd/research/object_detection/trainer.py", line 276, in train
train_config.prefetch_queue_capacity, data_augmentation_options)
File "/home/luiz/model_ssd/research/object_detection/trainer.py", line 59, in create_input_queue
tensor_dict = create_tensor_dict_fn()
File "train.py", line 125, in get_next
dataset_builder.build(config)).get_next()
File "/home/luiz/model_ssd/research/object_detection/builders/dataset_builder.py", line 134, in build
config.input_path[:], input_reader_config)
File "/home/luiz/model_ssd/research/object_detection/builders/dataset_builder.py", line 80, in read_dataset
sloppy=config.shuffle))
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 973, in apply
dataset = transformation_func(self)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/interleave_ops.py", line 76, in _apply_fn
buffer_output_elements, prefetch_input_elements)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/readers.py", line 130, in __init__
cycle_length, block_length)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1959, in __init__
super(InterleaveDataset, self).__init__(input_dataset, map_func)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1928, in __init__
self._map_func.add_to_graph(ops.get_default_graph())
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/function.py", line 484, in add_to_graph
self._create_definition_if_needed()
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/function.py", line 319, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/function.py", line 336, in _create_definition_if_needed_impl
outputs = self._func(*inputs)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1916, in tf_map_func
dataset = map_func(nested_args)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/readers.py", line 196, in __init__
filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1014, in convert_to_tensor
as_ref=False)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 947, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float32: 'Tensor("arg0:0", shape=(), dtype=float32)'

I did put the "prints" in the function above
def _TensorTensorConversionFunction(t, dtype=None, name=None, as_ref=False):
_ = name, as_ref
print('t',t,'dt', dtype ,'name', name,'ref', as_ref)
print('t.dtype',t.dtype)
print('comp', dtype.is_compatible_with(t.dtype))
if dtype and not dtype.is_compatible_with(t.dtype):
print('entry21',t, dtype , name, as_ref)
raise ValueError(
"Tensor conversion requested dtype %s for Tensor with dtype %s: %r" %
(dtype.name, t.dtype.name, str(t)))
return t

Tensor("global_step:0", shape=(), dtype=int64_ref, device=/device:CPU:0) dt None name ref ref True
t.dtype

print('comap', dtype.is_compatible_with(t.dtype))
AttributeError: 'NoneType' object has no attribute 'is_compatible_with'

lgribeiro on 7 Jul 2018

A complete answer: The issue is that step: 0 in the following block is not recognized by the newer releases:

schedule {
            step: 0
            learning_rate: .0001
          }

A fix is to delete this block and changing initial_learning_rate to the learning_rate described in the block. The file that you have to modify is the one that you pass as option --pipeline_config_path to the train.py script.

dshahrokhian on 2 Aug 2018

@pkulzc there are like 3 blocks for schedule, which one?

dscha09 on 17 Aug 2018

👍1

I am getting this error

(tensorflow1) C:\tensorflow1\models\research\object_detection>python train.py --
logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_in
ception_v2_pets.config
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\trainer.
py:260: create_global_step (from tensorflow.contrib.framework.python.ops.variabl
es) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 0 to match input file shards.

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 274, i
n train
train_config.prefetch_queue_capacity, data_augmentation_options)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 59, in
create_input_queue
tensor_dict = create_tensor_dict_fn()
File "train.py", line 121, in get_next
dataset_builder.build(config)).get_next()
File "C:\tensorflow1\models\research\object_detection\builders\dataset_builder
.py", line 186, in build
process_fn, config.input_path[:], input_reader_config)
File "C:\tensorflow1\models\research\object_detection\utils\dataset_util.py",
line 141, in read_dataset
sloppy=config.shuffle))
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 973, in apply
dataset = transformation_func(self)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\c
ontrib\data\python\ops\interleave_ops.py", line 76, in _apply_fn
buffer_output_elements, prefetch_input_elements)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 130, in init
cycle_length, block_length)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1959, in init
super(InterleaveDataset, self).init(input_dataset, map_func)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1928, in init
self._map_func.add_to_graph(ops.get_default_graph())
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 484, in add_to_graph
self._create_definition_if_needed()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 319, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 336, in _create_definition_if_needed_impl
outputs = self._func(*inputs)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1916, in tf_map_func
dataset = map_func(nested_args)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 196, in init
filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1014, in convert_to_tensor
as_ref=False)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 947, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float
32: 'Tensor("arg0:0", shape=(), dtype=float32)'

Can anyone explain why I am getting this error?

moinkhan3012 on 24 Sep 2018

I am getting this error

(tensorflow1) C:\tensorflow1\models\research\object_detection>python train.py --
logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_in
ception_v2_pets.config
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\trainer.
py:260: create_global_step (from tensorflow.contrib.framework.python.ops.variabl
es) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 0 to match input file shards.

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 274, i
n train
train_config.prefetch_queue_capacity, data_augmentation_options)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 59, in
create_input_queue
tensor_dict = create_tensor_dict_fn()
File "train.py", line 121, in get_next
dataset_builder.build(config)).get_next()
File "C:\tensorflow1\models\research\object_detection\builders\dataset_builder
.py", line 186, in build
process_fn, config.input_path[:], input_reader_config)
File "C:\tensorflow1\models\research\object_detection\utils\dataset_util.py",
line 141, in read_dataset
sloppy=config.shuffle))
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 973, in apply
dataset = transformation_func(self)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\c
ontrib\data\python\ops\interleave_ops.py", line 76, in _apply_fn
buffer_output_elements, prefetch_input_elements)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 130, in init
cycle_length, block_length)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1959, in init
super(InterleaveDataset, self).init(input_dataset, map_func)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1928, in init
self._map_func.add_to_graph(ops.get_default_graph())
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 484, in add_to_graph
self._create_definition_if_needed()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 319, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 336, in _create_definition_if_needed_impl
outputs = self._func(*inputs)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1916, in tf_map_func
dataset = map_func(nested_args)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 196, in init
filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1014, in convert_to_tensor
as_ref=False)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 947, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float
32: 'Tensor("arg0:0", shape=(), dtype=float32)'

Can anyone explain why I am getting this error?

moinkhan3012 on 24 Sep 2018

When I according to the suggestion of @pkulzc to remove that block, the problem can be solved, however, a new problem is followed, which is:

File "D:\Anaconda2\envs\tensorflow1\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in __call__
ret = func(*args)

File "D:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
self._metrics = self.evaluate()

File "D:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 212, in evaluate
all_metrics_per_category=self._all_metrics_per_category)

File "D:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 236, in ComputeMetrics
self.evaluate()

File "D:\tensorflow1\models\research\pycocotools\cocoeval.py", line 156, in evaluate
for catId in catIds

File "D:\tensorflow1\models\research\pycocotools\cocoeval.py", line 158, in
for imgId in p.imgIds

File "D:\tensorflow1\models\research\pycocotools\cocoeval.py", line 264, in evaluateImg
ious = self.ious[imgId, catId][:, gtind] if len(self.ious[imgId, catId]) > 0 else self.ious[imgId, catId]

TypeError: object of type 'NoneType' has no len()

I hope you can give me some ways to solve it, thanks very much!

GitWR on 30 Sep 2018

Hi, can someone help me with these errors?

Vinzia on 10 Mar 2019

I deleted the block :
schedule {
step: 0
learning_rate: .0002
}
but i found an other error "Attempting to fetch value instead of handling error Internal: no supported devices found for platform CUDA
Abandon (core dumped)"

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/platform/app.py:125: main (from __main__) is deprecated and will be removed in a future version.
Instructions for updating:
Use object_detection/model_main.py.
WARNING:tensorflow:From /home/user/great/model/research/object_detection/legacy/trainer.py:266: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /home/user/great/model/research/object_detection/builders/dataset_builder.py:80: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.experimental.parallel_interleave(...).
WARNING:tensorflow:From /home/user/great/model/research/object_detection/legacy/trainer.py:65: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/user/great/model/research/object_detection/core/batcher.py:96: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.batch(batch_size) (or padded_batch(...) if dynamic_pad=True).
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/input.py:753: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the tf.data module.
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/input.py:753: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the tf.data module.
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/image_ops_impl.py:1055: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py:1624: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From /home/user/great/model/research/object_detection/core/losses.py:345: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

/home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From /home/user/great/model/research/object_detection/meta_architectures/faster_rcnn_meta_arch.py:2298: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/depthwise_weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/pointwise_weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py:737: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py:737: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-06-06 15:50:48.647811: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2494300000 Hz
2019-06-06 15:50:48.648157: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x559ee428ed20 executing computations on platform Host. Devices:
2019-06-06 15:50:48.648256: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-06-06 15:50:48.868474: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-06 15:50:48.869202: I tensorflow/compiler/xla/service/platform_util.cc:194] StreamExecutor cuda device (0) is of insufficient compute capability: 3.5 required, device is 2.1
2019-06-06 15:50:48.869415: F tensorflow/stream_executor/lib/statusor.cc:34] Attempting to fetch value instead of handling error Internal: no supported devices found for platform CUDA
Abandon (core dumped)

monadhif on 6 Jun 2019

tessor flow custom training

ERROR:raise ValueError('First step cannot be zero.') ValueError: First step cannot be zero.

SOLUTION: object_detection\training\ .config

train_config: {
batch_size: 1
optimizer {
momentum_optimizer: {
learning_rate: {
manual_step_learning_rate {
initial_learning_rate: 0.0002
schedule {
step: 900000
learning_rate: .00002
}
schedule {
step: 1200000
learning_rate: .000002
}
}
}
momentum_optimizer_value: 0.9
}
use_moving_average: false
}

bharath5673 on 6 Jul 2019

Hello
In config file change "step: 0" to "step: 1 or non-zero value"

optimizer {
    momentum_optimizer {
      learning_rate {
        manual_step_learning_rate {
          initial_learning_rate: 0.000300000014249
          schedule `{`
            step: 1
            learning_rate: 0.000300000014249
          }

irfankilic on 12 Jun 2020

hello so here I try to train a model of mask r cnn on my own give with the command
python D:\Projet\CV-CommonTools\model_main.py /
--model_dir training /
--pipeline_config_path training\mask_rcnn_inception_v2.config /
--num_train_steps 60000 /
--sample_1_of_n_eval_examples=0 /
--alsologtostderr

but i get the following error

Traceback (most recent call last):
  File "D:\Projet\model_main.py", line 110, in <module>
    tf.app.run()
  File "C:\Users\Ibrahim\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\Ibrahim\AppData\Local\Programs\Python\Python36\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\Ibrahim\AppData\Local\Programs\Python\Python36\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "D:\Projet\model_main.py", line 71, in main
    FLAGS.sample_1_of_n_eval_on_train_examples))
TypeError: create_estimator_and_inputs() missing 1 required positional argument: 'hparams'

could someone guided me please