Models: [object_detection] Python 3 incompatibilities and other errors

Created on 3 Aug 2018 · 8Comments · Source: tensorflow/models

System information

What is the top-level directory of the model you are using: research/object_detection
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.9.0
Bazel version (if compiling from source): -
CUDA/cuDNN version: 9
GPU model and memory: GeForce GTX 1080 Ti
Exact command to reproduce: model_main.py --pipeline_config_path="samples/configs/ssd_mobilenet_v1_0.75_depth_quantized_300x300_pets_sync.config" --model_dir="data/pet-train" --logtostderr

Note that for running the script on Windows, you need the adapted pycocotools.

Describe the problem

Summary: I've tried to run the tutorial for training pets detection locally, but couldn't get it to run, due to numerous bugs.

The latest version of Tensorflow Object Detection API is incompatible with Python 3 and even following the official tutorial does not work appropriately, when trying to train locally.

I've encountered the following issues, when trying to clone the repo and run from master-branch (commit f0e10716):

Python 2 constructs, that are incompatible with Python 3. I've fixed two issues in this branch of my fork.
After fixing the Python 3 incompatibilities, training started, but both the --logtostderror (how it used to be) and the --alsologtostderror flags were simply ignored by the model_main.py script and no output was produced on the std-output (information to the tensorboard was written, though).
There is no clear commit/tag/branch, which would indicate the acclaimed July 13 release. I've asked that question also on SO.
Finally, it started to train but crashed when running the evaluation with a TypeError: can't pickle dict_values objects (see log below).

Source code / logs

2018-08-03 16:59:36.773937: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1306] Invalid argument: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in __call__
    ret = func(*args)

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_evaluation.py", line 339, in first_value_func
    self._metrics = self.evaluate()

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_evaluation.py", line 193, in evaluate
    self._detection_boxes_list)

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Programmieren\Anaconda3\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


Traceback (most recent call last):
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call
    return fn(*args)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in __call__
    ret = func(*args)

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_evaluation.py", line 339, in first_value_func
    self._metrics = self.evaluate()

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_evaluation.py", line 193, in evaluate
    self._detection_boxes_list)

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Programmieren\Anaconda3\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


     [[Node: PyFunc_1 = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_7", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
     [[Node: Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_25/NonMaxSuppressionV3/_4177 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_4912_...pressionV3", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Alex/Repositories/MusicObjectDetector-TF2/research/object_detection/model_main.py", line 101, in <module>
    tf.app.run()
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "C:/Users/Alex/Repositories/MusicObjectDetector-TF2/research/object_detection/model_main.py", line 97, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 447, in train_and_evaluate
    return executor.run()
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 531, in run
    return self.run_local()
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 681, in run_local
    eval_result, export_results = evaluator.evaluate_and_export()
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 886, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 460, in evaluate
    output_dir=self.eval_dir(name))
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1386, in _evaluate_run
    config=self._session_config)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\training\evaluation.py", line 212, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 689, in __exit__
    self._close_internal(exception_type)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 721, in _close_internal
    h.end(self._coordinated_creator.tf_sess)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 824, in end
    self._final_ops, feed_dict=self._final_ops_feed_dict)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 900, in run
    run_metadata_ptr)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
    run_metadata)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in __call__
    ret = func(*args)

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_evaluation.py", line 339, in first_value_func
    self._metrics = self.evaluate()

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_evaluation.py", line 193, in evaluate
    self._detection_boxes_list)

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Programmieren\Anaconda3\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


     [[Node: PyFunc_1 = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_7", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
     [[Node: Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_25/NonMaxSuppressionV3/_4177 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_4912_...pressionV3", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'PyFunc_1', defined at:
  File "C:/Users/Alex/Repositories/MusicObjectDetector-TF2/research/object_detection/model_main.py", line 101, in <module>
    tf.app.run()
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "C:/Users/Alex/Repositories/MusicObjectDetector-TF2/research/object_detection/model_main.py", line 97, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 447, in train_and_evaluate
    return executor.run()
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 531, in run
    return self.run_local()
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 681, in run_local
    eval_result, export_results = evaluator.evaluate_and_export()
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 886, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 453, in evaluate
    input_fn, hooks, checkpoint_path)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1348, in _evaluate_build_graph
    features, labels, model_fn_lib.ModeKeys.EVAL, self.config)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1107, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\model_lib.py", line 387, in model_fn
    include_metrics_per_category=eval_config.include_metrics_per_category)
  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\eval_util.py", line 629, in get_eval_metric_ops_for_evaluators
    input_data_fields.groundtruth_is_crowd)))
  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_evaluation.py", line 349, in get_estimator_eval_metric_ops
    first_value_op = tf.py_func(first_value_func, [], tf.float32)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\ops\script_ops.py", line 384, in py_func
    func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\ops\script_ops.py", line 227, in _internal_py_func
    input=inp, token=token, Tout=Tout, name=name)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_script_ops.py", line 130, in py_func
    "PyFunc", input=input, token=token, Tout=Tout, name=name)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
    op_def=op_def)
  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Programmieren\Anaconda3\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in __call__
    ret = func(*args)

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_evaluation.py", line 339, in first_value_func
    self._metrics = self.evaluate()

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_evaluation.py", line 193, in evaluate
    self._detection_boxes_list)

  File "C:\Users\Alex\Repositories\MusicObjectDetector-TF2\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Programmieren\Anaconda3\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


     [[Node: PyFunc_1 = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_7", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
     [[Node: Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_25/NonMaxSuppressionV3/_4177 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_4912_...pressionV3", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Source

apacha

👍4

Most helpful comment

I changed results.dataset['categories'] = copy.deepcopy(self.dataset['categories']) to results.dataset['categories'] = self.dataset['categories'], then it works for me.

yjfncu on 6 Aug 2018

👍3 ❤1

All 8 comments

Related: #4982

apacha on 3 Aug 2018

I also meet the same problems, how can you solve this problems?

yjfncu on 5 Aug 2018

👍1

I changed results.dataset['categories'] = copy.deepcopy(self.dataset['categories']) to results.dataset['categories'] = self.dataset['categories'], then it works for me.

yjfncu on 6 Aug 2018

👍3 ❤1

@yjfncu but the meaning is different without copy.deepcopy , is the result ok ?

hengshan123 on 7 Aug 2018

@hengshanji, yes,it is the right result for me, and i trained model on pet dataset by local, and trained model on my own dateset also, all this predicted result if very good for me. " copy.deepcopy", the code if this line means that give one dict value to another dict value ,so if you debug this code of line you will found the meaning is the same

yjfncu on 9 Aug 2018

You can also follow the advice here (https://github.com/tensorflow/models/issues/4780#issuecomment-405441448) and change https://github.com/tensorflow/models/blob/master/research/object_detection/model_lib.py#L390 to list(category_index.values())

gfkeith on 10 Aug 2018

👍1

And two more issues in object_detection_evaluation.py:

Unicode does not exist in Python 3.
The way this statement is written, gives an "Augmented assign to tuple literal or generator expression not possible" error.

apacha on 10 Aug 2018

Who can help me identify the input and output name in this graph? I need them as input and output parameters to convert my .pb file to .tflite file