Models: InvalidArgumentError: indices[0] = 0 is not in [0, 0) + Caught OutOfRangeError. Stopping Training

[chansekar]

Hi Guys,

When i try to train the model, i get the below error.

[[Node: prefetch_queue_Dequeue = QueueDequeueV2component_types=[DT_STRING, DT_STRING, DT_INT32, DT_INT32, DT_INT64, DT_INT32, DT_INT64, DT_INT32, DT_INT32, DT_STRING, DT_BOOL, DT_INT32, DT_FLOAT, DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT64, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"]]
INFO:tensorflow:Caught OutOfRangeError. Stopping Training.
INFO:tensorflow:Finished training! Saving model to disk.
Traceback (most recent call last):
File "object_detection/train.py", line 198, in
tf.app.run()
File "/home/diji/.virtualenvs/d3/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection/train.py", line 194, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/home/diji/MySSDExp/models-master/object_detection/trainer.py", line 296, in train
saver=saver)
File "/home/diji/.virtualenvs/d3/lib/python3.5/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 759, in train
sv.saver.save(sess, sv.save_path, global_step=sv.global_step)
File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
next(self.gen)
File "/home/diji/.virtualenvs/d3/lib/python3.5/site-packages/tensorflow/python/training/supervisor.py", line 964, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/home/diji/.virtualenvs/d3/lib/python3.5/site-packages/tensorflow/python/training/supervisor.py", line 792, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "/home/diji/.virtualenvs/d3/lib/python3.5/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/diji/.virtualenvs/d3/lib/python3.5/site-packages/six.py", line 686, in reraise
raise value
File "/home/diji/.virtualenvs/d3/lib/python3.5/site-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
enqueue_callable()
File "/home/diji/.virtualenvs/d3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1063, in _single_operation_run
target_list_as_strings, status, None)
File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
next(self.gen)
File "/home/diji/.virtualenvs/d3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0] = 0 is not in [0, 0)
[[Node: cond_2/RandomCropImage/PruneCompleteleyOutsideWindow/Gather/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_INT64, validate_indices=true, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_2/RandomCropImage/PruneCompleteleyOutsideWindow/Gather/Gather_1/Switch:1, cond_2/RandomCropImage/PruneCompleteleyOutsideWindow/Reshape)]]

I am not sure where i am going wrong, need help on it.

Regards
S.Chandrasekar

[pooyan1983]

Hi, have you found any solution to this?

[Baby47]

i also encounterd this problem .
indices[1] = 1 is not in [0, 1) + Caught OutOfRangeError

[dwSun]

Uninstall Tensorflow and reinstall it from pypi may fix this problem.
Make sure every time you compilation is clean. Delete everything from your last compilation.

[tanndx17]

Have you solved the problem? I got the same error.

[kznovo]

I had a similar issue but fixing the list elements (xmin, ymin, xmax, ymax, classes_text, classes) in TFRecords file solved the problem for me.
https://stackoverflow.com/a/49745801/9006027

[wt-huang]

Closing as this is resolved

[omar16100]

stuck in a similar issue and the stackoverflow answer isn't helping either

[tharindukprooftec]

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0] = 65386 is not in [0, 21824)

how to fix this

[Tanmay-Kulkarni101]

Has anyone got a fix to this issue. I tried running object detection with different models. It didn't work out and I am facing the same error. Can someone help me out?

[lucifer2288]

same issue in tf_agents/bandits

[Tanmay-Kulkarni101]

I got a solution to my issue. I have posted the solution here.