Mask_rcnn: Train the model on GPU and try to run inference mode on CPU. But got errors.

[cooleel]

InvalidArgumentError (see above for traceback): indices[42] = 65464 is not in [0, 65280)
     [[Node: ROI/Gather_2 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI/strided_slice_6, ROI/strided_slice_7)]]

The same script runs well on the GPU machine.
tensorflow-gpu ==1.12.0
Keras == 2.2.4

using the same versions of tensorflow and keras on a local mac pro(the cpu machine I want to run the inference mode).

Any ideas?
Thanks.

[linchundan88]

I encounter the same problem just as you. I am using tensorflow-gpu 12 and tensorflow-gpu 10.
However the models I trained some months ago(may be use an earlier TF version) run inference well both on GPU and CPU.

[linchundan88]

Ubuntu 16.
Error Msg:
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[19] = 36771 is not in [0, 36720)

[linchundan88]

I use both keras2.2.2 and keras 2.2.4 encounter this problem.

[cooleel]

I use both keras2.2.2 and keras 2.2.4 encounter this problem.

Yup, I have tried both keras 2.2.2 and 2.2.4, as well as different versions of tensorflow.
Any chance to recall the TF/keras version you have used before?
Thanks!

[cooleel]

@linchundan88 check it out: https://github.com/matterport/Mask_RCNN/issues/143
maybe this is the reason why the previous models worked.

[linchundan88]

Tf 1.9 and tf2.2.4 were used before, and it worked fine.
I will check out #143 later.

[deepdiy]

Are you using the default ResNet-101 with FPN structure? If so, you have a mismatch in the number of elements between RPN_ANCHOR_SCALES and BACKBONE_STRIDES. Also, you have RPN_ANCHOR_SCALES defined twice.

In general, if you are using ResNet-101 with FPN and taking the C4 backbone, there are 5 levels in the FPN. Thus, you should have 5 elements in RPN_ANCHOR_SCALES and 5 elements in BACKBONE_STRIDES.

[cooleel]

Are you using the default ResNet-101 with FPN structure? If so, you have a mismatch in the number of elements between RPN_ANCHOR_SCALES and BACKBONE_STRIDES. Also, you have RPN_ANCHOR_SCALES defined twice.

In general, if you are using ResNet-101 with FPN and taking the C4 backbone, there are 5 levels in the FPN. Thus, you should have 5 elements in RPN_ANCHOR_SCALES and 5 elements in BACKBONE_STRIDES.

Hi, thank you for your reply. Actually, I was using Resnet50, and at first, there was a mismatch between RPN_ANCHOR_SCALES and BACKBONE_STRIDES, later I lined up these two, but still not working. I might try ResNet-101.

[cooleel]

It's working! @deepdiy Thanks!

[Trotts]

Hi @deepdiy @cooleel, I am also getting this error and was wondering if you could clarify what is required to fix this issue? My model was trained on a GPU originally, however I am now running in inference mode on a CPU.

My model's config can be found here. As can be seen, my BACKBONE_STRIDES are [4, 8, 16, 32, 64], and my RPN_ANCHOR_SCALES are (16, 32, 64, 128, 256). Am I right in thinking that to get this running on a CPU, these must be the same (and if so why does this not matter on a GPU), or am I misunderstanding what I need to do to fix this?

Thank you for your help in advance!

[PROFRAGU]

Configurations:
BACKBONE resnet101
BACKBONE_STRIDES [4, 8, 16, 32, 64]
BATCH_SIZE 8
BBOX_STD_DEV [0.1 0.1 0.2 0.2]
COMPUTE_BACKBONE_SHAPE None
DETECTION_MAX_INSTANCES 100
DETECTION_MIN_CONFIDENCE 0.7
DETECTION_NMS_THRESHOLD 0.3
FPN_CLASSIF_FC_LAYERS_SIZE 1024
GPU_COUNT 1
GRADIENT_CLIP_NORM 5.0
IMAGES_PER_GPU 8
IMAGE_CHANNEL_COUNT 3
IMAGE_MAX_DIM 256
IMAGE_META_SIZE 24
IMAGE_MIN_DIM 128
IMAGE_MIN_SCALE 0
IMAGE_RESIZE_MODE square
IMAGE_SHAPE [256 256 3]
LEARNING_MOMENTUM 0.9
LEARNING_RATE 0.001
LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE 14
MASK_SHAPE [28, 28]
MAX_GT_INSTANCES 100
MEAN_PIXEL [123.7 116.8 103.9]
MINI_MASK_SHAPE (56, 56)
NAME shapes
NUM_CLASSES 12
POOL_SIZE 7
POST_NMS_ROIS_INFERENCE 1000
POST_NMS_ROIS_TRAINING 2000
PRE_NMS_LIMIT 6000
ROI_POSITIVE_RATIO 0.33
RPN_ANCHOR_RATIOS [0.5, 1, 2]
RPN_ANCHOR_SCALES (8, 16, 32, 64, 128)
RPN_ANCHOR_STRIDE 1
RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD 0.7
RPN_TRAIN_ANCHORS_PER_IMAGE 256
STEPS_PER_EPOCH 100
TOP_DOWN_PYRAMID_SIZE 256
TRAIN_BN False
TRAIN_ROIS_PER_IMAGE 32
USE_MINI_MASK True
USE_RPN_ROIS True
VALIDATION_STEPS 5
WEIGHT_DECAY 0.0001

[PROFRAGU]

InvalidArgumentError Traceback (most recent call last)
in ()
6 learning_rate=config.LEARNING_RATE,
7 epochs=1,
----> 8 layers='heads')

7 frames
/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in __call__(self, args, *kwargs)
1470 ret = tf_session.TF_SessionRunCallable(self._session._session,
1471 self._handle, args,
-> 1472 run_metadata_ptr)
1473 if run_metadata:
1474 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

InvalidArgumentError: indices[51] = [195, 1216546] does not index into param shape [256,12,4]
[[{{node mrcnn_bbox_loss/GatherNd}}]]

[PROFRAGU]

How to fix..I am running my code using google colab..pls help me