Mask_rcnn: Unpredictable behaviour when doing prediction

[bylowerik]

I am trying to do some predictions using the Mask_RCNN network. The goal is to use it on my own images. I used the samples/coco/coco.py script as inpiration. I loop through the image names and do prediction on each image. The problem is that it crashes, sometimes before the first image and sometimes after 4 images. I do not understand why and the error message is not clear to me.

This comes when batch_outs = f(ins_batch) is called in training_arrays.py line 294.

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[1] = 93 is not in [0, 4) [[{{node mrcnn_detection/map/while/GatherV2_2}} = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mrcnn_detection/map/while/strided_slice, mrcnn_detection/map/while/non_max_suppression/NonMaxSuppressionV3, mrcnn_detection/map/while/PadV2/paddings/0/0)]]

Seems to me it is related to TensorFlow?

Another error message that comes frequently is:
./tensorflow/core/framework/tensor.h:643] Check failed: new_num_elements == NumElements() (1 vs. 2)

and

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0] = 32 is not in [0, 15) [[{{node mrcnn_detection/map/while/GatherV2_2}} = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mrcnn_detection/map/while/strided_slice, mrcnn_detection/map/while/non_max_suppression/NonMaxSuppressionV3, mrcnn_detection/map/while/PadV2/paddings/0/0)]]

I have tried both with and without GPU, but no difference.

I have also tried the script under samples/coco/coco.py and that gives the same error.

My code

import os
import sys
import numpy as np
import cv2 as cv
import matplotlib.pyplot as plt
ROOT_DIR = os.path.abspath("/home/erikbylow/Code/LocalLibs/Mask_RCNN/")
sys.path.append("/home/erikbylow/Code/LocalLibs/Mask_RCNN/mrcnn/")
sys.path.append("/home/erikbylow/Code/LocalLibs/Mask_RCNN/")

from mrcnn.config import Config
from mrcnn import model as modellib, utils
MODEL_PATH = os.path.join(ROOT_DIR, "masks/mask_rcnn_coco.h5")
IMAGE_PATH = os.path.join(ROOT_DIR, "images/")


class CocoConfig(Config):
    """Configuration for training on MS COCO.
    Derives from the base Config class and overrides values specific
    to the COCO dataset.
    """
    # Give the configuration a recognizable name
    NAME = "coco"

    # We use a GPU with 12GB memory, which can fit two images.
    # Adjust down if you use a smaller GPU.
    IMAGES_PER_GPU = 1

    # Uncomment to train on 8 GPUs (default is 1)
    # GPU_COUNT = 8

    # Number of classes (including background)
    NUM_CLASSES = 1 + 80  # COCO has 80 classes


def load_images(path):
    """ Load images in IMAGE_PATH """
    dir_list = os.listdir(path)
    return dir_list

def predict_mask(im_names, im_path, model):
    res = []
    print("Start Prediction...")
    for name in im_names:
        im_name = os.path.join(im_path, name)
        im = cv.imread(im_name, cv.IMREAD_COLOR)
        im = cv.cvtColor(im, cv.COLOR_BGR2RGB)
        r = model.detect([im], verbose=1)[0]
        mask = r["masks"]
        tmp_img = mask[:,:,0]
        tmp_img.astype(np.int)
        plt.figure(0)
        plt.imshow(tmp_img)
        plt.figure(1)
        plt.imshow(im)
        plt.show()
        print(im.shape)

    print("... Prediction Done")

def create_model(config, model_dir, weight_path):
    model = modellib.MaskRCNN(mode='inference', config=config, model_dir=model_dir)
    model.load_weights(weight_path, by_name=True)
    return model

def get_config():
    class InferenceConfig(CocoConfig):
        # Set batch size to 1 since we'll be running inference on
        # one image at a time. Batch size = GPU_COUNT * IMAGES_PER_GPU
        GPU_COUNT = 1
        IMAGES_PER_GPU = 1
        DETECTION_MIN_CONFIDENCE = 0
    config = InferenceConfig()
    return config


config = get_config()
model_dir = "/home/erikbylow/Code/logs/"
weight_path = os.path.join(ROOT_DIR, "masks/mask_rcnn_coco.h5")

model = create_model(config, model_dir, weight_path)

im_names = load_images(IMAGE_PATH)
predict_mask(im_names, IMAGE_PATH, model)

[jonatanwulcan]

Could this be related: https://github.com/tensorflow/tensorflow/issues/22750

Try downgrading to TF 1.10.0.

[ppwwyyxx]

This is exactly https://github.com/tensorflow/tensorflow/issues/22750

[jonatanwulcan]

I had this problem as well. Downgrading to TF 1.10.0 did not help.

I ended up disabling interOp multithreading for TF, that seems to have solved the problem for now.

[fabioaraujopt]

I had this problem as well. Downgrading to TF 1.10.0 did not help.

I ended up disabling interOp multithreading for TF, that seems to have solved the problem for now.

how did you do that?

[jonatanwulcan]

I set inter_op_parallelism_threads to 1 in the tf config proto. You pass in
the config proto when you create the tf session.

Im not sure how to do it when using keras to wrap tf.

https://www.tensorflow.org/performance/performance_guide#optimizing_for_cpu

On Wed, 28 Nov 2018 at 10:00, fabioaraujopt [e-mail]
wrote:

I had this problem as well. Downgrading to TF 1.10.0 did not help.

I ended up disabling interOp multithreading for TF, that seems to have
solved the problem for now.

how did you do that?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/matterport/Mask_RCNN/issues/1002#issuecomment-442370192,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADXOFlbC16frQxucb5N7uRoSeUF16n-Dks5uzlCRgaJpZM4XFrym
.

[fabioaraujopt]

I run demo.ipynb where should I change that parameter?
Thanks in advance

[jonatanwulcan]

Try putting this somewhere in the beginning. I didn't test this code in any
way

import tensorflow
import keras.backend
config = tensorflow.ConfigProto()
config.inter_op_parallelism_threads = 1
keras.backend.set_session(tensorflow.Session(config=config))

On Wed, Nov 28, 2018 at 11:34 AM fabioaraujopt [e-mail]
wrote:

I run demo.ipynb where should I change that parameter?
Thanks in advance

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/matterport/Mask_RCNN/issues/1002#issuecomment-442399549,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADXOFuzhvRbB4eicvIMxccCc2YG0b8q0ks5uzmbAgaJpZM4XFrym
.

[fabioaraujopt]

i did the configs are defined in the following class:

class InferenceConfig(coco.CocoConfig):
GPU_COUNT = 1
IMAGES_PER_GPU = 1

is that a parameter that i should place here?

[jonatanwulcan]

No, it's another config.

Try copy pasting the code I posted and put it somewhere in the beginning of
the file.

On Wed, Nov 28, 2018 at 2:17 PM fabioaraujopt [e-mail]
wrote:

i did the configs are defined in the following class:

class InferenceConfig(coco.CocoConfig):
GPU_COUNT = 1
IMAGES_PER_GPU = 1

is that a parameter that i should place here?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/matterport/Mask_RCNN/issues/1002#issuecomment-442443253,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADXOFhQHkaSI8lJJTo5B5AkOqh6R2YAkks5uzoz9gaJpZM4XFrym
.

[JonathanLehner]

seems to work with that

[tayo]

This is exactly tensorflow/tensorflow#22750

This issue was closed on October 18. Is this still showing up at HEAD?

If anyone has a small reproducer, that would be appreciated.

[eselkin]

I get the error in use and random crashes when doing prediction.
On HEAD but using TF 1.12

Setting the TF config to use one thread does solve the issue.

[anjali-chadha]

Try putting this somewhere in the beginning. I didn't test this code in any way import tensorflow import keras.backend config = tensorflow.ConfigProto()
config.inter_op_parallelism_threads = 1
keras.backend.set_session(tensorflow.Session(config=config))

I faced a similar issue and code snippet by @jonatanwulcan solves the problem. Thanks!