Models: [deeplab][mobilenetV3] ValueError '%s is not decorated with @add_arg_scope' on training

[limonene51]

I faced a problem on the training of deeplab with mobilenetV3 models.
Only when using mobilenetV3, the error occured and could not be resolved yet.
I wonder if you could tell me how to solve the problem.

Thank you for your help in advance

Conditions:
OS: Mac OS Mojave 10.14.6
Python: 3.6.8
Tensorflow: 1.14.0
tf-slim: 1.2.0

What I tried.
1, Clone the models repo (commit:b3ef7ae9)
2, Download and convert ADE20K with bash script
3, Prepare initial_checkpoint (mobilenetv3_large_cityscapes_trainfine and xception65_coco_voc_trainval from deeplab model zoo)
4, Run training according to the "Running DeepLab on ADE20K Semantic Segmentation Dataset"
with following commands

For Xception:(Succeeded)
python train.py --logtostderr --training_number_of_steps=30 --train_split="train" --model_variant="xception_65" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --train_crop_size="513,513" --train_batch_size=4 --min_resize_value=513 --max_resize_value=513 --resize_factor=16 --dataset="ade20k" --tf_initial_checkpoint=datasets/deeplabv3_pascal_trainval/model.ckpt --train_logdir=datasets/trained_checkpoints/testv3 --dataset_dir=datasets/ADE20K/tfrecord --initialize_last_layer=false

For mobilenetV3:(Failed)
python train.py --logtostderr --training_number_of_steps=30 --train_split="train" --model_variant="mobilenet_v3_large_seg" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --train_crop_size="513,513" --train_batch_size=4 --min_resize_value=513 --max_resize_value=513 --resize_factor=16 --dataset="ade20k" --tf_initial_checkpoint=datasets/deeplab_mnv3_large_cityscapes_trainfine/model.ckpt --train_logdir=datasets/trained_checkpoints/testv3 --dataset_dir=datasets/ADE20K/tfrecord --initialize_last_layer=false --image_pooling_crop_size=769,769 --image_pooling_stride=4,5 --add_image_level_feature=1 --aspp_convs_filters=128 --aspp_with_concat_projection=0 --aspp_with_squeeze_and_excitation=1 --decoder_use_sum_merge=1 --decoder_filters=19 --decoder_output_is_logits=1 --image_se_uses_qsigmoid=1 --decoder_output_stride=8 --output_stride=32 --image_pyramid=1

Bold arguments were added according to the model zoo comments and some issues.

Results:
Only when using mobilenetV3, the value error occurred and could not train the model.
The trace back is as shown below:

Traceback (most recent call last):
File "train.py", line 464, in
tf.app.run()
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 321, in main
clones = model_deploy.create_clones(config, model_fn, args=model_args)
File "/Users/XXX/version_control/models/research/slim/deployment/model_deploy.py", line 192, in create_clones
outputs = model_fn(args, *kwargs)
File "train.py", line 252, in _build_deeplab
'total_training_steps': FLAGS.training_number_of_steps,
File "/Users/XXX/version_control/models/research/deeplab/model.py", line 323, in multi_scale_logits
nas_training_hyper_parameters=nas_training_hyper_parameters)
File "/Users/XXX/version_control/models/research/deeplab/model.py", line 577, in _get_logits
nas_training_hyper_parameters=nas_training_hyper_parameters)
File "/Users/XXX/version_control/models/research/deeplab/model.py", line 404, in extract_features
use_bounded_activation=model_options.use_bounded_activation)
File "/Users/XXX/version_control/models/research/deeplab/core/feature_extractor.py", line 649, in extract_features
weight_decay=weight_decay)
File "/Users/XXX/version_control/models/research/deeplab/core/feature_extractor.py", line 308, in mobilenet_v2_arg_scope
[conv_blocks.expanded_conv], normalizer_fn=slim.batch_norm), \
File "/Users/XXX/.pyenv/versions/3.6.8/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 152, in arg_scope
_name_op(op))
ValueError: ('%s is not decorated with @add_arg_scope', ('nets.mobilenet.conv_blocks', 'expanded_conv'))

[alrikai]

I've seen the same error when exporting the deeplab_mnv3_large_cityscapes_trainfine pretrained model with export_model.py. Downgrading tf-slim from 1.1.0 to 1.0.0 seems to "fix" this error for me.

[limonene51]

alrikai, I appreciate you so much.
Your advise for downgrading tf-slim could resolved the value error!

Unfortunately, another error occurred during training as show below.
Because the 44943436 is a multiple of 4 x 796 x 796 x 19 and the setting argument --decode_filter = 19, I tried to change the decode_filter = 151.
The training worked well but I think it is not good way.
I wonder if you could tell me how to resolve the error correctly.

--------error message---------
I0618 13:19:12.596302 4485174720 session_manager.py:500] Running local_init_op.
I0618 13:19:12.758799 4485174720 session_manager.py:502] Done running local_init_op.
I0618 13:19:17.617907 4485174720 learning.py:754] Starting Session.
I0618 13:19:17.632869 123145438388224 supervisor.py:1117] Saving checkpoint to path datasets/trained_checkpoints/testv3/model.ckpt
I0618 13:19:17.633080 4485174720 learning.py:768] Starting Queues.
I0618 13:19:26.413588 123145433133056 supervisor.py:1099] global_step/sec: 0
I0618 13:19:31.727333 123145427877888 coordinator.py:219] Error reported to Coordinator: Input to reshape is a tensor with 44943436 values, but the requested shape requires a multiple of 151
[[node Reshape_1 (defined at /Users/XXX/version_control/models/research/deeplab/utils/train_utils.py:119) ]]

Errors may have originated from an input operation.
Input Source operations connected to node Reshape_1:
ResizeBilinear_2 (defined at /Users/XXX/version_control/models/research/deeplab/utils/train_utils.py:92)

Original stack trace for 'Reshape_1':
File "train.py", line 464, in
tf.app.run()
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 321, in main
clones = model_deploy.create_clones(config, model_fn, args=model_args)
File "/Users/XXX/version_control/models/research/slim/deployment/model_deploy.py", line 192, in create_clones
outputs = model_fn(args, *kwargs)
File "train.py", line 270, in _build_deeplab
scope=output)
File "/Users/XXX/version_control/models/research/deeplab/utils/train_utils.py", line 119, in add_softmax_cross_entropy_loss_for_each_scale
logits = tf.reshape(logits, shape=[-1, num_classes])
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 7715, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
Traceback (most recent call last):
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 44943436 values, but the requested shape requires a multiple of 151
[[{{node Reshape_1}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 495, in run
self.run_loop()
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 1045, in run_loop
[self._sv.summary_op, self._sv.global_step])
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 44943436 values, but the requested shape requires a multiple of 151
[[node Reshape_1 (defined at /Users/XXX/version_control/models/research/deeplab/utils/train_utils.py:119) ]]

Errors may have originated from an input operation.
Input Source operations connected to node Reshape_1:
ResizeBilinear_2 (defined at /Users/XXX/version_control/models/research/deeplab/utils/train_utils.py:92)

Original stack trace for 'Reshape_1':
File "train.py", line 464, in
tf.app.run()
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 321, in main
clones = model_deploy.create_clones(config, model_fn, args=model_args)
File "/Users/XXX/version_control/models/research/slim/deployment/model_deploy.py", line 192, in create_clones
outputs = model_fn(args, *kwargs)
File "train.py", line 270, in _build_deeplab
scope=output)
File "/Users/XXX/version_control/models/research/deeplab/utils/train_utils.py", line 119, in add_softmax_cross_entropy_loss_for_each_scale
logits = tf.reshape(logits, shape=[-1, num_classes])
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 7715, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()

Traceback (most recent call last):
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 44943436 values, but the requested shape requires a multiple of 151
[[{{node Reshape_1}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 1004, in managed_session
yield sess
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 775, in train
train_step_kwargs)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 490, in train_step
run_metadata=run_metadata)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 44943436 values, but the requested shape requires a multiple of 151
[[node Reshape_1 (defined at /Users/XXX/version_control/models/research/deeplab/utils/train_utils.py:119) ]]

Errors may have originated from an input operation.
Input Source operations connected to node Reshape_1:
ResizeBilinear_2 (defined at /Users/XXX/version_control/models/research/deeplab/utils/train_utils.py:92)

Original stack trace for 'Reshape_1':
File "train.py", line 464, in
tf.app.run()
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 321, in main
clones = model_deploy.create_clones(config, model_fn, args=model_args)
File "/Users/XXX/version_control/models/research/slim/deployment/model_deploy.py", line 192, in create_clones
outputs = model_fn(args, *kwargs)
File "train.py", line 270, in _build_deeplab
scope=output)
File "/Users/XXX/version_control/models/research/deeplab/utils/train_utils.py", line 119, in add_softmax_cross_entropy_loss_for_each_scale
logits = tf.reshape(logits, shape=[-1, num_classes])
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 7715, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 464, in
tf.app.run()
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 458, in main
save_interval_secs=FLAGS.save_interval_secs)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 790, in train
ignore_live_threads=ignore_live_threads)
File "/Users/XXX/.pyenv/versions/3.6.8/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 1014, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 839, in stop
ignore_live_threads=ignore_live_threads)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 495, in run
self.run_loop()
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 1045, in run_loop
[self._sv.summary_op, self._sv.global_step])
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 44943436 values, but the requested shape requires a multiple of 151
[[node Reshape_1 (defined at /Users/XXX/version_control/models/research/deeplab/utils/train_utils.py:119) ]]

Errors may have originated from an input operation.
Input Source operations connected to node Reshape_1:
ResizeBilinear_2 (defined at /Users/XXX/version_control/models/research/deeplab/utils/train_utils.py:92)

Original stack trace for 'Reshape_1':
File "train.py", line 464, in
tf.app.run()
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 321, in main
clones = model_deploy.create_clones(config, model_fn, args=model_args)
File "/Users/XXX/version_control/models/research/slim/deployment/model_deploy.py", line 192, in create_clones
outputs = model_fn(args, *kwargs)
File "train.py", line 270, in _build_deeplab
scope=output)
File "/Users/XXX/version_control/models/research/deeplab/utils/train_utils.py", line 119, in add_softmax_cross_entropy_loss_for_each_scale
logits = tf.reshape(logits, shape=[-1, num_classes])
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 7715, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/Users/XXX/.pyenv/versions/3.6.8/envs/common368/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()

[limonene51]

The problem seemed resolved by myself.
The argument --decorde_filter must be same with num_classes of dataset.

Thank you for your kind cooperation

[bmabir17]

I have also faced similar error. Down grading to tf_slim v1.0 have solved it