Mask_rcnn: InvalidArgumentError

[Telang8100]

Hi
I am trying to run the following cell

import warnings
warnings.filterwarnings("ignore")
model.train(dataset_train, dataset_val,learning_rate=config.LEARNING_RATE, epochs=NUM_EPOCHS, layers='all',augmentation=augmentation)

And I am getting this error

File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)

InvalidArgumentError: indices[553] = 960 is not in [0, 960)
[[Node: ROI_4/GatherV2_20 = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI_4/strided_slice_42, ROI_4/strided_slice_43, training_5/SGD/gradients/roi_align_classifier_4/concat_grad/mod)]]

Caused by op 'ROI_4/GatherV2_20', defined at:
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\spyder\utils\ipython\start_kernel.py", line 269, in
main()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\spyder\utils\ipython\start_kernel.py", line 265, in main
kernel.start()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\kernelapp.py", line 486, in start
self.io_loop.start()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tornado\platform\asyncio.py", line 127, in start
self.asyncio_loop.run_forever()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\asyncio\base_events.py", line 422, in run_forever
self._run_once()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\asyncio\base_events.py", line 1432, in _run_once
handle._run()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\asyncio\events.py", line 145, in _run
self._callback(self._args)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tornado\ioloop.py", line 759, in _run_callback
ret = callback()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tornado\stack_context.py", line 276, in null_wrapper
return fn(args, *kwargs)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 263, in enter_eventloop
self.eventloop(self)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\eventloops.py", line 134, in loop_qt5
return loop_qt4(kernel)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\eventloops.py", line 122, in loop_qt4
_loop_qt(kernel.app)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\eventloops.py", line 106, in loop_qt
app.exec()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\eventloops.py", line 39, in process_stream_events
kernel.do_one_iteration()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 298, in do_one_iteration
stream.flush(zmq.POLLIN, 1)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 357, in flush
self._handle_recv()
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 480, in _handle_recv
self._run_callback(callback, msg)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 432, in _run_callback
callback(args, *kwargs)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tornado\stack_context.py", line 276, in null_wrapper
return fn(args, *kwargs)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 233, in dispatch_shell
handler(stream, idents, msg)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 208, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\ipykernel\zmqshell.py", line 537, in run_cell
return super(ZMQInteractiveShell, self).run_cell(args, *kwargs)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2662, in run_cell
raw_cell, store_history, silent, shell_futures)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2785, in _run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2909, in run_ast_nodes
if self.run_code(code, result):
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2963, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
runfile('E:/Spyder/pneumonia22.py', wdir='E:/Spyder')
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "E:/Spyder/pneumonia22.py", line 209, in
model = modellib.MaskRCNN(mode='training', config=config, model_dir=MODEL_DIR)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\mask_rcnn-2.1-py3.6.egg\mrcnn\model.py", line 1845, in init
self.keras_model = self.build(mode=mode, config=config)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\mask_rcnn-2.1-py3.6.egg\mrcnn\model.py", line 1973, in build
config=config)([rpn_class, rpn_bbox, anchors])
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\keras\engine\topology.py", line 617, in call
output = self.call(inputs, *kwargs)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\mask_rcnn-2.1-py3.6.egg\mrcnn\model.py", line 296, in call
names=["pre_nms_anchors"])
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\mask_rcnn-2.1-py3.6.egg\mrcnn\utils.py", line 829, in batch_slice
output_slice = graph_fn(*inputs_slice)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\mask_rcnn-2.1-py3.6.egg\mrcnn\model.py", line 294, in
pre_nms_anchors = utils.batch_slice([anchors, ix], lambda a, x: tf.gather(a, x),
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tensorflow\python\ops\array_ops.py", line 2666, in gather
return gen_array_ops.gather_v2(params, indices, axis, name=name)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 3760, in gather_v2
"GatherV2", params=params, indices=indices, axis=axis, name=name)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
op_def=op_def)
File "C:\Users\SIDDHESHWAR\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): indices[553] = 960 is not in [0, 960)
[[Node: ROI_4/GatherV2_20 = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI_4/strided_slice_42, ROI_4/strided_slice_43, training_5/SGD/gradients/roi_align_classifier_4/concat_grad/mod)]]

[drctrue]

Hello! How did you get rid of this error? I think I ran into the same problem. Maybe you can share your experience? Thank you!

[Telang8100]

Comment out the following line in the DetectorConfig function.

RPN_ANCHOR_SCALES = (32, 64)
Should work then.

[Nimrods]

Hi,
I tried to make the network work with only 3 scales of bbox (64,128,256)
So I Changed : RPN_ANCHOR_SCALES = (64, 128, 256)
and i get the following error:
File "/home/nimrod/nanit/deepalgo/mask_net/test_frcnn.py", line 414, in main
results = test.run_on_dataset(dirs, iou_thresh)
File "/home/nimrod/nanit/deepalgo/mask_net/test_frcnn.py", line 303, in run_on_dataset
prediction = self.model.detect([image])[0]
File "/home/nimrod/nanit/deepalgo/mask_net/Mask_RCNN/mrcnn/model.py", line 2558, in detect
self.keras_model.predict([molded_images, image_metas, anchors], verbose=0)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1835, in predict
verbose=verbose, steps=steps)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1331, in _predict_loop
batch_outs = f(ins_batch)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2482, in __call__
**self.session_kwargs)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[41] = 25366 is not in [0, 25200)
[[Node: ROI/GatherV2_2 = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI/strided_slice_6, ROI/strided_slice_7, mrcnn_mask_deconv/add/y)]]

Caused by op 'ROI/GatherV2_2', defined at:
File "/home/nimrod/nanit/deepalgo/mask_net/test_frcnn.py", line 423, in
main(SegmentationTasks.head)
File "/home/nimrod/nanit/deepalgo/mask_net/test_frcnn.py", line 398, in main
test = Test(task, trained_model_dir, dataset_name=dataset_name)
File "/home/nimrod/nanit/deepalgo/mask_net/test_frcnn.py", line 262, in __init__
model_dir=self.config.pre_trained_model_dir)
File "/home/nimrod/nanit/deepalgo/mask_net/Mask_RCNN/mrcnn/model.py", line 1855, in __init__
self.keras_model = self.build(mode=mode, config=config)
File "/home/nimrod/nanit/deepalgo/mask_net/Mask_RCNN/mrcnn/model.py", line 1984, in build
config=config)([rpn_class, rpn_bbox, anchors])
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/keras/engine/topology.py", line 619, in __call__
output = self.call(inputs, *kwargs)
File "/home/nimrod/nanit/deepalgo/mask_net/Mask_RCNN/mrcnn/model.py", line 300, in call
names=["pre_nms_anchors"])
File "/home/nimrod/nanit/deepalgo/mask_net/Mask_RCNN/mrcnn/utils.py", line 829, in batch_slice
output_slice = graph_fn(inputs_slice)
File "/home/nimrod/nanit/deepalgo/mask_net/Mask_RCNN/mrcnn/model.py", line 298, in
pre_nms_anchors = utils.batch_slice([anchors, ix], lambda a, x: tf.gather(a, x),
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 2666, in gather
return gen_array_ops.gather_v2(params, indices, axis, name=name)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3141, in gather_v2
"GatherV2", params=params, indices=indices, axis=axis, name=name)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/home/nimrod/nanit/deepalgo/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): indices[41] = 25366 is not in [0, 25200)
[[Node: ROI/GatherV2_2 = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI/strided_slice_6, ROI/strided_slice_7, mrcnn_mask_deconv/add/y)]]

Any clues on how to proceed?

[MartinezLaura]

I'm having the same error, any solution on this?

Thanks in advance!

[linhaojia13]

I meet a similar error：

---

InvalidArgumentError Traceback (most recent call last)
in
19 # Pick a set of random images
20 image_ids = np.random.choice(dataset.image_ids, 10)
---> 21 APs = compute_batch_ap(image_ids)
22 print("mAP @ IoU=50: ", np.mean(APs))

in compute_batch_ap(image_ids)
8 image_id, use_mini_mask=False)
9 # Run object detection
---> 10 results = model.detect([image], verbose=0)
11 # Compute AP
12 r = results[0]

~/temp/src_to_use/Mask_RCNN/mrcnn/model.py in detect(self, images, verbose)
2522 # Run object detection
2523 detections, _, _, mrcnn_mask, _, _, _ =\
-> 2524 self.keras_model.predict([molded_images, image_metas, anchors], verbose=0)
2525 # Process detections
2526 results = []

~/.local/lib/python3.5/site-packages/keras/engine/training.py in predict(self, x, batch_size, verbose, steps)
1167 batch_size=batch_size,
1168 verbose=verbose,
-> 1169 steps=steps)
1170
1171 def train_on_batch(self, x, y,

~/.local/lib/python3.5/site-packages/keras/engine/training_arrays.py in predict_loop(model, f, ins, batch_size, verbose, steps)
292 ins_batch[i] = ins_batch[i].toarray()
293
--> 294 batch_outs = f(ins_batch)
295 batch_outs = to_list(batch_outs)
296 if batch_index == 0:

~/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py in __call__(self, inputs)
2713 return self._legacy_call(inputs)
2714
-> 2715 return self._call(inputs)
2716 else:
2717 if py_any(is_tensor(x) for x in inputs):

~/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py in _call(self, inputs)
2673 fetched = self._callable_fn(array_vals, run_metadata=self.run_metadata)
2674 else:
-> 2675 fetched = self._callable_fn(array_vals)
2676 return fetched[:len(self.outputs)]
2677

~/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py in __call__(self, args, *kwargs)
1437 ret = tf_session.TF_SessionRunCallable(
1438 self._session._session, self._handle, args, status,
-> 1439 run_metadata_ptr)
1440 if run_metadata:
1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)
526 None, None,
527 compat.as_text(c_api.TF_Message(self.status.status)),
--> 528 c_api.TF_GetCode(self.status.status))
529 # Delete the underlying status object from memory otherwise it stays alive
530 # as there is a reference to status from this from the traceback due to

InvalidArgumentError: indices[0] = 2 is not in [0, 1)
[[{{node mrcnn_detection/map/while/GatherV2_2}} = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mrcnn_detection/map/while/strided_slice, mrcnn_detection/map/while/non_max_suppression/NonMaxSuppressionV3, mrcnn_detection/map/while/PadV2/paddings/0/0)]]

Do you have any clue on this? Thank you!

[jasonfghx]

it's good

[cooleel]

Met the same error when trying to run the inference mode on the CPU.
The same script runs well on a GPU machine.
Any ideas?
Thanks.

[Alejandro-Valdes]

@cooleel, did you figure out how to fix this, I've got the same problem

[mercutiodesign]

Has anyone figured out the solution to this? I just encountered the same error here.

[cooleel]

Yes, I finally fixed this. Check out #1291 @Alejandro-Valdes @mercutiodesign

[ChaoLi977]

It works when I comment out the "RPN_ANCHOR_SCALES = (32, 64)" line.

[masterika]

Same problem. Trying to skip smaller scales to get 10x speed because I know my objects are on a larger scale.

[shamim4901]

Same problem. Seeking kind assist to fix up the problem !!

InvalidArgumentError Traceback (most recent call last)
~/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, args)
1326 try:
-> 1327 return fn(args)
1328 except errors.OpError as e:

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
1311 return self._call_tf_sessionrun(
-> 1312 options, feed_dict, fetch_list, target_list, run_metadata)
1313

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata)
1419 self._session, options, feed_dict, fetch_list, target_list,
-> 1420 status, run_metadata)
1421

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)
515 compat.as_text(c_api.TF_Message(self.status.status)),
--> 516 c_api.TF_GetCode(self.status.status))
517 # Delete the underlying status object from memory otherwise it stays alive

InvalidArgumentError: indices[0] = 172820 is not in [0, 16368)
[[Node: ROI/Gather_2 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI/strided_slice_6, ROI/strided_slice_7)]]

During handling of the above exception, another exception occurred:

InvalidArgumentError Traceback (most recent call last)
in
1 dataset_dir = "/home/ubuntu/work/unet/dataset/com_syn_img/"
----> 2 train(model, dataset_dir, subset="train")

in train(model, dataset_dir, subset)
32 epochs=20,
33 augmentation=augmentation,
---> 34 layers='heads')
35
36 print("Train all layers")

~/work/mrcnn/mrcnn/model.py in train(self, train_dataset, val_dataset, learning_rate, epochs, layers, augmentation, custom_callbacks, no_augmentation_sources)
2372 max_queue_size=100,
2373 workers=workers,
-> 2374 use_multiprocessing=True,
2375 )
2376 self.epoch = max(self.epoch, epochs)

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/keras/legacy/interfaces.py in wrapper(args, *kwargs)
85 warnings.warn('Update your ' + object_name + 86 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87 return func(args, *kwargs)
88 wrapper._original_function = func
89 return wrapper

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
2040 outs = self.train_on_batch(x, y,
2041 sample_weight=sample_weight,
-> 2042 class_weight=class_weight)
2043
2044 if not isinstance(outs, list):

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/keras/engine/training.py in train_on_batch(self, x, y, sample_weight, class_weight)
1760 ins = x + y + sample_weights
1761 self._make_train_function()
-> 1762 outputs = self.train_function(ins)
1763 if len(outputs) == 1:
1764 return outputs[0]

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py in __call__(self, inputs)
2271 updated = session.run(self.outputs + [self.updates_op],
2272 feed_dict=feed_dict,
-> 2273 **self.session_kwargs)
2274 return updated[:len(self.outputs)]
2275

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
903 try:
904 result = self._run(None, fetches, feed_dict, options_ptr,
--> 905 run_metadata_ptr)
906 if run_metadata:
907 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1138 if final_fetches or final_targets or (handle and feed_dict_tensor):
1139 results = self._do_run(handle, final_targets, final_fetches,
-> 1140 feed_dict_tensor, options, run_metadata)
1141 else:
1142 results = []

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1319 if handle is None:
1320 return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1321 run_metadata)
1322 else:
1323 return self._do_call(_prun_fn, handle, feeds, fetches)

~/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1338 except KeyError:
1339 pass
-> 1340 raise type(e)(node_def, op, message)
1341
1342 def _extend_graph(self):

InvalidArgumentError: indices[0] = 172820 is not in [0, 16368)
[[Node: ROI/Gather_2 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI/strided_slice_6, ROI/strided_slice_7)]]

Caused by op 'ROI/Gather_2', defined at:
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/traitlets/config/application.py", line 664, in launch_instance
app.start()
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 612, in start
self.io_loop.start()
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 149, in start
self.asyncio_loop.run_forever()
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/asyncio/base_events.py", line 442, in run_forever
self._run_once()
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/asyncio/base_events.py", line 1462, in _run_once
handle._run()
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/asyncio/events.py", line 145, in _run
self._callback(self._args)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tornado/ioloop.py", line 690, in
lambda f: self._run_callback(functools.partial(callback, future))
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tornado/gen.py", line 787, in inner
self.run()
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 365, in process_one
yield gen.maybe_future(dispatch(args))
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 268, in dispatch_shell
yield gen.maybe_future(handler(stream, idents, msg))
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 545, in execute_request
user_expressions, allow_stdin,
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 306, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 536, in run_cell
return super(ZMQInteractiveShell, self).run_cell(args, *kwargs)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2867, in run_cell
raw_cell, store_history, silent, shell_futures)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2895, in _run_cell
return runner(coro)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 68, in _pseudo_sync_runner
coro.send(None)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3072, in run_cell_async
interactivity=interactivity, compiler=compiler, result=result)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3263, in run_ast_nodes
if (await self.run_code(code, result, async_=asy)):
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
model = modellib.MaskRCNN(mode="training", config=config, model_dir=LOGS_DIR)
File "/home/ubuntu/work/mrcnn/mrcnn/model.py", line 1837, in __init__
self.keras_model = self.build(mode=mode, config=config)
File "/home/ubuntu/work/mrcnn/mrcnn/model.py", line 1965, in build
config=config)([rpn_class, rpn_bbox, anchors])
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/keras/engine/topology.py", line 602, in __call__
output = self.call(inputs, *kwargs)
File "/home/ubuntu/work/mrcnn/mrcnn/model.py", line 296, in call
names=["pre_nms_anchors"])
File "/home/ubuntu/work/mrcnn/mrcnn/utils.py", line 840, in batch_slice
output_slice = graph_fn(inputs_slice)
File "/home/ubuntu/work/mrcnn/mrcnn/model.py", line 294, in
pre_nms_anchors = utils.batch_slice([anchors, ix], lambda a, x: tf.gather(a, x),
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 2698, in gather
params, indices, validate_indices=validate_indices, name=name)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2672, in gather
validate_indices=validate_indices, name=name)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3290, in create_op
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/rcnn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1654, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): indices[0] = 172820 is not in [0, 16368)
[[Node: ROI/Gather_2 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI/strided_slice_6, ROI/strided_slice_7)]]

Same problem. Seeking kind assist to fix up the problem !!