Models: model_main.py does not work with own dataset, created by using old guide

Created on 17 Aug 2018 · 5Comments · Source: tensorflow/models

System information

What is the top-level directory of the model you are using: ssd_mobilenet_v1_coco_2018_01_28
OS Platform and Distribution: Ubuntu 16.04
TensorFlow installed from: source
TensorFlow version: 1.9
Bazel version: 0.15.0
CUDA/cuDNN version: 9.2 / 7.1.4
GPU model and memory: GTX 1080

Describe the problem

I made my own dataset using that and similar examples: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/using_your_own_dataset.md
My directory for using TF Object Detection API looks like this:

+ ~/recognition
    + data
        - label_map.pbtxt
        - train.record
        - eval.record
    + models
        + ssd_mobilenet_v1_coco

I have 3 classes and I indicated this in data/label_map.pbtxt and models/models/pipeline.config.
I used these commands and it work with my dataset:

python object_detection/legacy/train.py --logtostderr --pipeline_config_path=path/to/ssd_mobilenet_v1_coco/pipeline.config --train_dir=path/to/ssd_mobilenet_v1_coco/train

CUDA_VISIBLE_DEVICES="" python object_detection/legacy/eval.py --checkpoint_dir=path/to/ssd_mobilenet_v1_coco/train --logtostderr --pipeline_config_path=path/to/ssd_mobilenet_v1_coco/pipeline.config --eval_dir=path/to/ssd_mobilenet_v1_coco/eval

But when i try to use:
python object_detection/model_main.py \ --pipeline_config_path=${PIPELINE_CONFIG_PATH} \ --model_dir=${MODEL_DIR} \ --num_train_steps=${NUM_TRAIN_STEPS} \ --num_eval_steps=${NUM_EVAL_STEPS} \ --alsologtostderr

i get errors that will be listed below. I use python 3 (tried 3.5 and 3.6) with virtualenv.
Do I understand correctly that using_your_own_dataset-example is obsolete and now i need to create my own dataset a little differently? Where can I get a new guide?

Source code / logs

/home/user/tf_dir/models/research/object_detection/utils/visualization_utils.py:25: UserWarning:
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called before pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was originally set to 'TkAgg' by the following code:
File "/home/user/tf_dir/models/research/object_detection/model_main.py", line 26, in
from object_detection import model_lib
File "/home/user/tf_dir/models/research/object_detection/model_lib.py", line 26, in
from object_detection import eval_util
File "/home/user/tf_dir/models/research/object_detection/eval_util.py", line 28, in
from object_detection.metrics import coco_evaluation
File "/home/user/tf_dir/models/research/object_detection/metrics/coco_evaluation.py", line 20, in
from object_detection.metrics import coco_tools
File "/home/user/tf_dir/models/research/object_detection/metrics/coco_tools.py", line 47, in
from pycocotools import coco
File "/home/user/tf_dir/models/research/pycocotools/coco.py", line 49, in
import matplotlib.pyplot as plt
File "/home/user/.local/lib/python3.6/site-packages/matplotlib/pyplot.py", line 71, in
from matplotlib.backends import pylab_setup
File "/home/user/.local/lib/python3.6/site-packages/matplotlib/backends/__init__.py", line 16, in
line for line in traceback.format_stack()

import matplotlib; matplotlib.use('Agg') # pylint: disable=multiple-statements
WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x7f759cd7ad90>) includes params argument, but params are not passed to Estimator.
2018-08-17 20:30:47.773766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-08-17 20:30:47.774241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.8095
pciBusID: 0000:08:00.0
totalMemory: 7.92GiB freeMemory: 7.19GiB
2018-08-17 20:30:47.774256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2018-08-17 20:30:47.950557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-17 20:30:47.950593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2018-08-17 20:30:47.950603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2018-08-17 20:30:47.950801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6942 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:08:00.0, compute capability: 6.1)
Traceback (most recent call last):
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [24] rhs shape= [546]
[[Node: save/Assign_6 = Assign[T=DT_FLOAT, _class=["loc:@BoxPredictor_1/ClassPredictor/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](BoxPredictor_1/ClassPredictor/biases, save/RestoreV2/_13)]]
[[Node: save/RestoreV2/_46 = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_52_save/RestoreV2", _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/RestoreV2:23)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/tf_dir/models/research/object_detection/model_main.py", line 101, in
tf.app.run()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/user/tf_dir/models/research/object_detection/model_main.py", line 97, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 447, in train_and_evaluate
return executor.run()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 531, in run
return self.run_local()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 681, in run_local
eval_result, export_results = evaluator.evaluate_and_export()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 886, in evaluate_and_export
hooks=self._eval_spec.hooks)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 460, in evaluate
output_dir=self.eval_dir(name))
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1386, in _evaluate_run
config=self._session_config)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/evaluation.py", line 209, in _evaluate_once
session_creator=session_creator, hooks=hooks) as session:
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 826, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 549, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1012, in __init__
_WrappedSession.__init__(self, self._create_session())
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1017, in _create_session
return self._sess_creator.create_session()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 706, in create_session
self.tf_sess = self._session_creator.create_session()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 477, in create_session
init_fn=self._scaffold.init_fn)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/session_manager.py", line 281, in prepare_session
config=config)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/session_manager.py", line 195, in _restore_checkpoint
saver.restore(sess, checkpoint_filename_with_path)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1752, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [24] rhs shape= [546]
[[Node: save/Assign_6 = Assign[T=DT_FLOAT, _class=["loc:@BoxPredictor_1/ClassPredictor/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](BoxPredictor_1/ClassPredictor/biases, save/RestoreV2/_13)]]
[[Node: save/RestoreV2/_46 = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_52_save/RestoreV2", _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/RestoreV2:23)]]

Caused by op 'save/Assign_6', defined at:
File "/home/user/tf_dir/models/research/object_detection/model_main.py", line 101, in
tf.app.run()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/user/tf_dir/models/research/object_detection/model_main.py", line 97, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 447, in train_and_evaluate
return executor.run()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 531, in run
return self.run_local()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 681, in run_local
eval_result, export_results = evaluator.evaluate_and_export()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 886, in evaluate_and_export
hooks=self._eval_spec.hooks)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 460, in evaluate
output_dir=self.eval_dir(name))
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1386, in _evaluate_run
config=self._session_config)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/evaluation.py", line 209, in _evaluate_once
session_creator=session_creator, hooks=hooks) as session:
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 826, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 549, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1012, in __init__
_WrappedSession.__init__(self, self._create_session())
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1017, in _create_session
return self._sess_creator.create_session()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 706, in create_session
self.tf_sess = self._session_creator.create_session()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 468, in create_session
self._scaffold.finalize()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 212, in finalize
self._saver = training_saver._get_saver_or_default() # pylint: disable=protected-access
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 856, in _get_saver_or_default
saver = Saver(sharded=True, allow_empty=True)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1284, in __init__
self.build()
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1296, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1333, in _build
build_save=build_save, build_restore=build_restore)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 775, in _build_internal
restore_sequentially, reshape)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 453, in _AddShardedRestoreOps
name="restore_shard"))
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 422, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 113, in restore
self.op.get_shape().is_fully_defined())
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 219, in assign
validate_shape=validate_shape)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
use_locking=use_locking, name=name)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [24] rhs shape= [546]
[[Node: save/Assign_6 = Assign[T=DT_FLOAT, _class=["loc:@BoxPredictor_1/ClassPredictor/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](BoxPredictor_1/ClassPredictor/biases, save/RestoreV2/_13)]]
[[Node: save/RestoreV2/_46 = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_52_save/RestoreV2", _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/RestoreV2:23)]]

Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in
import apt
File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/user/py_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [24] rhs shape= [546]
[[Node: save/Assign_6 = Assign[T=DT_FLOAT, _class=["loc:@BoxPredictor_1/ClassPredictor/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](BoxPredictor_1/ClassPredictor/biases, save/RestoreV2/_13)]]
[[Node: save/RestoreV2/_46 = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_52_save/RestoreV2", _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/RestoreV2:23)]]

During handling of the above exception, another exception occurred:

awaiting response

Source

kulsemig

All 5 comments

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
Exact command to reproduce

tensorflowbutler on 17 Aug 2018

👎4 😕1

Hi -- Thanks for reaching out.

I'd recommend Stack overflow (https://stackoverflow.com/questions/tagged/tensorflow) as a place to reach out. This is a great resource for problems such as yours, this issue queue is intended for bugs in the particular model themselves.

Thanks,
Victor

bitfort on 17 Aug 2018

Stackoverflow turned out to be ineffective. In #5083 same problem was solved, but it did not help me. Can you update you guides for creating own dataset, recomended directory structure and running model_main.py? Because it looks like they are obsolete.

kulsemig on 27 Aug 2018

Okay! This is the same issue that I had. It would be a help if there were a guidance on identifying source of this error

Manish-rai21bit on 28 Aug 2018

I met the same issue,too.But I have not got any errors except some warnings below
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7fad19e90268>) includes params argument, but params are not passed to Estimator. WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. 2018-08-28 15:49:35.874786: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2018-08-28 15:49:38.472203: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2018-08-28 15:49:38.472682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: name: Tesla M40 24GB major: 5 minor: 2 memoryClockRate(GHz): 1.112
Nothing has happened until now,I have waited 20 minutes,no output string print on screen,it seems like program stops running(yet it's not)