Mask_rcnn: How to convert Mask rcnn model to Tensorflow .pb

[luoshanwei]

I want to run the mask rcnn on android , but I have not .pb file

[ps48]

@luoshanwei You'll need to export the keras checkpoint to tensorflow .pb file and then use it.

[ericj974]

@ps48 This will be possible once #167 is complete no (python layer not serializable) ?

[ps48]

@ericj974 Yes, it is complete I had tried exporting the file. It works! just waiting for the merge.

[Surmeh]

Hello, Could you send me the script you used to convert the checkpoint to .pb? I tried using https://github.com/amir-abdi/keras_to_tensorflow/blob/master/keras_to_tensorflow.py but it gives "ValueError: No model found in config file."

[ericj974]

@Surmeh Code here

[Surmeh]

Thanks alot @ericj974. What does "inference_config" refer to in here?
def export(inference_config,
train_log_dirpath,
model_path = None):

[ericj974]

I should have commented the function. It's a config instance used for inference (being related to the config used for training, albeit not exactly the same). Look at train_shapes.ipynb for an example of it.

[whrenstone]

@luoshanwei have you found the solution? i found a error that tell me that inputs are mismatch. Could you give me a suggestion or a sample code? Many thanks!

[Cpruce]

I can verify that https://github.com/amir-abdi/keras_to_tensorflow/blob/master/keras_to_tensorflow.py works for extracting the tensorflow model

[Adriel-M]

@Cpruce what are you feeding into that script? I'm trying to feed in the saved model from model.keras_model.save(path) and I'm getting hit with lambda t: tf.reshape(t, [tf.shape(t)[0], -1, 2]))(x) NameError: name 'tf' is not defined

Also I'm getting ValueError: Unknown layer: BatchNorm if I don't comment out the BatchNorm layers.

EDIT:

So I bypassed NameError: name 'tf' is not defined by converting the lambda function as a regular function and importing tensorflow. Now I need to figure out how to handle the custom layers (ProposalLayer, etc.)

[Cpruce]

python3 keras_to_tensorflow.py -input_model_file saved_model_mrcnn_eval -output_model_file model.pb -num_outputs=7

I do model.keras_model.save(path) in coco.py as well. I definitely hit the BatchNorm and, if I remember correctly, was not able to resolve the issue on my laptop (I mainly tried getting the dep versions up-to-date). Not a great answer but I think what worked for me was just doing it on my linux machine.

[jmtatsch]

@Cpruce where exactly in coco.py do you save the model?
I tried after evaluate_coco in line 516 because the graph should be built up fully after that but keras_to_tensorflow complains that the architecture is not contained in the .h5 file. I also attempted to export the model after compile in model.py line 2083 but then I get deepcopy errors, maximum recursion depth exceeded.

[Surmeh]

Hi @Cpruce, we tried to extract the MaskRCNN weights to a model file, but failed to do so. Will you be able to share the model(.pb) for it, if you have successfully got it?

[Cpruce]

@jmtatsch I exported the model on line 516 as well. Are you doing model.keras_model.save(path)?

@Surmeh I tried zipping it but it's still too big. pb file size is 249M and github only allows 10M.

[Surmeh]

@Cpruce : Alright, thanks for trying.

[Cpruce]

@Surmeh can you please share the command you're running and the error?

[ps48]

@Cpruce to have the exported file in github, we can upload it as a binary file in the next release (similar to model releases).

[Cpruce]

@ps48 sounds like a good idea 👍 Only downside is he won't be able to make changes and then save the model, though that may not be necessary for his use case

[Surmeh]

@ps48 It would be really great if you could upload the binary file. Also, when are you planning your next release?

[Cpruce]

@waleedka

[jmtatsch]

@Cpruce would you be so kind and look up which keras/tf version you are running? and if your export is still working?

pip3 show keras
Version: 2.1.4

pip3 show tensorflow-gpu
Version: 1.4.0

[Cpruce]

Sure thing:

pip show keras
Name: Keras
Version: 2.1.2

pip show tensorflow-gpu
Name: tensorflow-gpu
Version: 1.3.0

The export still works. Which OS are you using?

[jmtatsch]

@Cpruce I am running Ubuntu 16.04 now with your respective keras and tensorflow-gpu versions but keras_to_tensorflow.py is still unable to load_model() the exported model from the model.h5. Seems as if my keras model.save(path) is unable to save the whole model to h5. I will try to make a minimal example to narrow this down further.

Nevertheless how exactly do you trigger your model export?
I use:
python3 coco.py evaluate --dataset=$MSCOCO_DATASET --model=coco

[Cpruce]

@jmtatsch
To produce h5:
python3 coco.py evaluate --dataset=$COCO_PATH --model=coco

To save model in coco.py:
evaluate_coco(model, dataset_val, coco, "bbox", limit=int(args.limit)) model.keras_model.save("mrcnn_eval.h5")

Extracting pb from h5:
python3 keras_to_tensorflow.py -input_model_file saved_model_mrcnn_eval.h5 -output_model_file model.pb -num_outputs=7

Could you paste the full stacktrace?

[Surmeh]

Hi @Cpruce
I am running the following command:
python3 coco.py evaluate --dataset=/home/surabhi/Tensorflow_Models/coco/val2014 --model=coco

Error:
Traceback (most recent call last):
File "coco.py", line 469, in
model.load_weights(model_path, by_name=True)
File "/home/surabhi/Tensorflow_Models/model.py", line 2037, in load_weights
topology.load_weights_from_hdf5_group_by_name(f, layers)
File "/home/surabhi/tensorflow/lib/python3.5/site-packages/keras/engine/topology.py", line 3260, in load_weights_from_hdf5_group_by_name
' element(s).')
ValueError: Layer #9 (named "res2a_branch2b") expects 0 weight(s), but the saved weights have 2 element(s).

[jmtatsch]

@Cpruce Saving and reloading a minimal model works in the same workspace, starting to run out of ideas here. Here is my full stack trace:

python3 keras_to_tensorflow.py -input_model_file mrcnn_eval.h5 -output_model_file model.pb -num_outputs=7                              tatsch@knecht2
usage: keras_to_tensorflow.py [-h] [-input_fld INPUT_FLD]
                              [-output_fld OUTPUT_FLD]
                              [-input_model_file INPUT_MODEL_FILE]
                              [-output_model_file OUTPUT_MODEL_FILE]
                              [-output_graphdef_file OUTPUT_GRAPHDEF_FILE]
                              [-num_outputs NUM_OUTPUTS]
                              [-graph_def GRAPH_DEF]
                              [-output_node_prefix OUTPUT_NODE_PREFIX]
                              [-quantize QUANTIZE]
                              [-theano_backend THEANO_BACKEND] [-f F]

set input arguments

optional arguments:
  -h, --help            show this help message and exit
  -input_fld INPUT_FLD
  -output_fld OUTPUT_FLD
  -input_model_file INPUT_MODEL_FILE
  -output_model_file OUTPUT_MODEL_FILE
  -output_graphdef_file OUTPUT_GRAPHDEF_FILE
  -num_outputs NUM_OUTPUTS
  -graph_def GRAPH_DEF
  -output_node_prefix OUTPUT_NODE_PREFIX
  -quantize QUANTIZE
  -theano_backend THEANO_BACKEND
  -f F
input args:  Namespace(f=None, graph_def=False, input_fld='.', input_model_file='mrcnn_eval.h5', num_outputs=7, output_fld='', output_graphdef_file='model.ascii', output_model_file='model.pb', output_node_prefix='output_node', quantize=False, theano_backend=False)
/home/tatsch/.virtualenvs/maskrcnn/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Input file specified (mrcnn_eval.h5) only holds the weights, and not the model defenition.
    Save the model using mode.save(filename.h5) which will contain the network architecture
    as well as its weights. 
    If the model is saved using model.save_weights(filename.h5), the model architecture is 
    expected to be saved separately in a json format and loaded prior to loading the weights.
    Check the keras documentation for more details (https://keras.io/getting-started/faq/)
Traceback (most recent call last):
  File "keras_to_tensorflow.py", line 123, in <module>
    raise err
  File "keras_to_tensorflow.py", line 114, in <module>
    net_model = load_model(weight_file_path)
  File "/home/tatsch/.virtualenvs/maskrcnn/lib/python3.5/site-packages/keras/models.py", line 240, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/home/tatsch/.virtualenvs/maskrcnn/lib/python3.5/site-packages/keras/models.py", line 314, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/home/tatsch/.virtualenvs/maskrcnn/lib/python3.5/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/tatsch/.virtualenvs/maskrcnn/lib/python3.5/site-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object
    list(custom_objects.items())))
  File "/home/tatsch/.virtualenvs/maskrcnn/lib/python3.5/site-packages/keras/engine/topology.py", line 2490, in from_config
    process_layer(layer_data)
  File "/home/tatsch/.virtualenvs/maskrcnn/lib/python3.5/site-packages/keras/engine/topology.py", line 2476, in process_layer
    custom_objects=custom_objects)
  File "/home/tatsch/.virtualenvs/maskrcnn/lib/python3.5/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/tatsch/.virtualenvs/maskrcnn/lib/python3.5/site-packages/keras/utils/generic_utils.py", line 134, in deserialize_keras_object
    ': ' + class_name)
ValueError: Unknown layer: BatchNorm

[Adriel-M]

@jmtatsch take a look at: https://github.com/matterport/Mask_RCNN/issues/218#issuecomment-365069480

I think the issue here is it can't export custom layers.

[Cpruce]

@Surmeh can you load the weights from the original h5 file?
@jmtatsch Im not sure if I've tried extracting the model outside of the workspace. Can you extract the pb there and then use the tf model somewhere else?

[jmtatsch]

@Cpruce Sorry by workspace I meant virtual environment. I also tried to run keras_to_tensorflow.py in the Mask_RCNN folder, without success.
@Adriel-M So you commented all BatchNorm's in model.py? Wouldn't that mess up the results? Or did you refactor it to use KL.BatchNormalization again?

[jmtatsch]

Ok, I am stuck at the ProposalLayer now as well.

[Surmeh]

@Cpruce nope, I can't.
Here is the code I'm running:
MODEL_DIR = os.path.join(ROOT_DIR, "logs")
model_path = os.path.join(ROOT_DIR, "mask_rcnn_coco.h5")
config = coco.CocoConfig()
class InferenceConfig(config.__class__):
# Set batch size to 1 since we'll be running inference on
# one image at a time. Batch size = GPU_COUNT * IMAGES_PER_GPU
GPU_COUNT = 1
IMAGES_PER_GPU = 1
DETECTION_MIN_CONFIDENCE = 0
config = InferenceConfig()
model = modellib.MaskRCNN(mode="inference", model_dir=MODEL_DIR,config=config)
model.load_weights(model_path)

Error: Its the same error as before, in this the layer name is not printed.
ValueError: You are trying to load a weight file containing 233 layers into a model with 1256 layers.

[Cpruce]

@jmtatsch what's wrong at the ProposalLayer?
@Surmeh you're trying with a clean version of the latest master and mask_rcnn_coco.h5?

[Surmeh]

@Cpruce I've got the saved model using the coco.py after getting the clean version of master. Now I will try to convert it to .pb using the keras_to_tensorflow script.

[chenyuZha]

Hello, I finally converted .h5 to .pb with the code of @ericj974 . Now I tried to do the inferences with .pb, but I had some problems about how to process the input images to feed in placeholder.

As I know, you should give 2 tensors to feed in placeholder, which are input_image:0 and image_meta:0.

I'm a little confused about how to get the tensors correspond placeholder image_meta, and I check in the model.py , the function detect, but I still not get the point..

Has anyone get idea here?

[Cpruce]

@chenyuZha You can use this, right?

def compose_image_meta(image_id, image_shape, window, active_class_ids):
    """Takes attributes of an image and puts them in one 1D array.
    image_id: An int ID of the image. Useful for debugging.
    image_shape: [height, width, channels]
    window: (y1, x1, y2, x2) in pixels. The area of the image where the real
            image is (excluding the padding)
    active_class_ids: List of class_ids available in the dataset from which
        the image came. Useful if training on images from multiple datasets
        where not all classes are present in all datasets.
    """
    meta = np.array(
        [image_id] +            # size=1
        list(image_shape) +     # size=3
        list(window) +          # size=4 (y1, x1, y2, x2) in image cooredinates
        list(active_class_ids)  # size=num_classes
    )
    return meta

[chenyuZha]

@Cpruce yes, but I'm not very sure about the steps of processing. So in my opinion,

1. Use function resize_image( in the script util)to get the parameters windows and molded_images.

2.Use function compose_image_meta to obtain the image_meta.

1. So then the molded_images correspond to the placeholder input_image:0 and image_meta correspond to placeholder image_meta:0.
2. sess.run to get nodes[detections, mrcnn_class, mrcnn_bbox, mrcnn_mask, rois, rpn_class, rpn_bbox]
   5.run the function unmold_detections to obtain finally [rois,class_ids,scores,masks] as Keras did.

Please tell me if it's correct.Thanks

[Cpruce]

Yup, sounds correct to me. If you're implementing in another environment, you'll probably have to debug each step...

[ashgolzar]

@chenyuZha Even if you use the right inputs, you will get an error due to tf.py_func in DetectionLayer.

"N.B. The tf.py_func() operation has the following known limitations:

The body of the function (i.e. func) will not be serialized in a GraphDef. Therefore, you should not use this function if you need to serialize your model and restore it in a different environment.

"
so you need to rewrite DetectionLayer using tensors instead of np ndarray

[Cpruce]

@ashgolzar

Seems like you're not on the latest master https://github.com/matterport/Mask_RCNN/pull/167

[ashgolzar]

@Cpruce apparently not :D
That's great :+1: I checked with the latest PR and it works fine.

[Cpruce]

@ashgolzar Awesome :)

[ivshli]

@ashgolzar you are using TF model in C++ environnement?

[ashgolzar]

@ivshli No I use Python.

[joelteply]

New here. Glad to see the progress. Let's see if we can get this working on C++. I'll try to help, also with reducing weight sizes. That's usually pretty straightforward. I have a shell script I can upload. My primary concern here, as it was with PSPNet, was with the BN nodes. That one also had a bunch of lamdas for the pyramiding.

[fastlater]

@ericj974 Thanks for the script. It does work.
I just want to add: in case someone want to inspect the exported model and check the placeholder names . You can use
import_pb_to_tensorboard.pyfrom here
python import_pb_to_tensorboard.py --model_dir=model/ownmodel.pb --log_dir=logsFolder

[liangbo-1]

@fastlater I've trained the model and saved it into the.H5 file. and then ,How should I use @ericj974's script to get the .pb file. What do I need to change in the script? thank you!

[fastlater]

What do I need to change in the script?

@liangbo-1 Nothing. Did you try it? Did you get error?
I am inspecting the exported model right now.

[liangbo-1]

@fastlater sorry, I don't know how to run the script and how to add my model ? Can you give me some specific guidance?

[liangbo-1]

@chenyuZha @ericj974 @fastlater Did you get the .pb file successfully with @ericj974's script? I have the following results，
python3 export_model.py -input_model_file 'mask_rcnn_ec_0002.h5' -output_model_file 'model.pb' -graph_def=True /usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype fromfloattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Using TensorFlow backend.

and Can you tell me how you did it?thank you

[ypflll]

@chenyuZha @joelteply I'm also working on using the .pb file in python or c++. It will be great help if you can share your code.

[fastlater]

@ericj974 Will you pull your script? I guess it will be of interest for several users who would like to serve their trained models.

@liangbo-1 As I mentioned before, code is doing fine without error. You got no error, you got a warning. Use the print function to check what the script outputs every few lines.

[ypflll]

Hi, I tried to use this code to run the .pb model file which I get using @ericj974 's code and follow the step @chenyuZha mentioned above:

output_graph_path = r"./mask_rcnn.pb"
with tf.Session() as sess:
    tf.global_variables_initializer().run()
    output_graph_def = tf.GraphDef()
    with open(output_graph_path, "rb") as f:
        output_graph_def.ParseFromString(f.read())
        _ = tf.import_graph_def(output_graph_def, name="")

    image = cv2.imread('/xxx/a.jpg')
    image = cv2.resize(image,(256,256))
    image = np.expand_dims(image,0)

    image_resized, window, scale,_ = resize_image(image, min_dim=None, max_dim=None, padding=False)
    image_meta = compose_image_meta(image_id=1, image_shape=image_resized.shape, window=window, active_class_ids=[0,1])
    image_meta = np.expand_dims(image_meta,0)
    graph = tf.get_default_graph()
    input_image = graph.get_tensor_by_name("input_image:0")
    input_image_meta = graph.get_tensor_by_name("input_image_meta:0")
    mrcnn_mask = graph.get_tensor_by_name("output_mrcnn_mask:0")

    #[detections, mrcnn_class, mrcnn_bbox, mrcnn_mask,rois, rpn_class, rpn_bbox]
    feed = {input_image:image_resized, input_image_meta:image_meta}
    result = sess.run(mrcnn_mask, feed_dict = feed)
    print(result.shape)

But I get a problem: I can only get a tensor in 'trainning' mode, (that is the code below: if mode == "training" ).
When I tried to get a tensor in 'inference' mode using code like: KL.Lambda(lambda x: x * 1, name="output_rois")(rois) or tf.identity, it always raises an error:
KeyError: "The name 'mask:0' refers to a Tensor which does not exist. The operation, 'mask', does not exist in the graph."

So I need to input a para that tells the model to do 'inference' mode? But how?

[fastlater]

@ypflll I dont know what is wrong with your code but you can follow this script. I will try to pull this after a few changes. Any suggestion about the code and how to make it looks cleaner is welcome.
I am waiting for @ericj974 because he wrote the first part, model serving and this could be considered as a second part, infering from a pb.

[ypflll]

@fastlater Good job. I'll try it.

[chenyuZha]

@fastlater
Thanks for your code. I tested it and it works well. Just a little question: I noticed that your use library scipy to do the pre-processing of images. (scipy.misc.imresize). I changed it to cv2.resize, and I obtained the masks which are little bit different( just the border), and the scores of each object are also different, I compare these 2 results and found that cv2.resize, en general, give a better detection scores.. As I used my own dataset, I don't know if it is the same case of yours...

[fastlater]

@ypflll You are welcome. Let me know if you get any trouble running the script.
@chenyuZha Thank you for your feedback. Actually, I just reused the resize function from the original utils.py. However, if you already tested both functions (scipy.misc.imresize and cv2.resize) and found cv2.resize function was giving better results, it sounds interesting.

[ypflll]

@fastlater Hi, your code works well.
But I have a problem with the time consuming. When predicting more than one pictures with size of 320*320, on a gpu, the first pic takes more than 60s!!! But the next pics only takes less than 300ms.

Have you tested your time cost and met this problem?

[fastlater]

@ypflll well, first image always take long than the rest. The slowness in the first run probably stems from things like device memory copying, cache misses, etc. As I said, the functions related to calculate, unmold and display results are taken from the demo. This problem also happens when you infer loading the model from weights, and not from pb. I have no idea if there is a way to speed up the first run, I hope it will.

I haven't calculate the time cost but my first run takes less than 60 seconds for sure.

[ypflll]

@fastlater I just measure the time cost of self.keras_model.predict(), so I don't think the problem relates to the pre-process or the post-process. Time cost depends on your image size, I use 320*320 and get this result.

I also tested in an old version code(1c51787), the first pic only takes 1.9s.
So, I think there is something with the last few commits. Maybe this is beyond this issue, I'll open another one.

[fastlater]

@ypflll Yes, open a new thread will be better. As I said, I noticed that even when testing the demo code so something is making the first run slower.

[githubXiaosong]

@fastlater Thank you for sharing your script. but I've used your script to make the following error:
Tensor("input_image:0", shape=(?, 1024, 1024, 3), dtype=float32)
Tensor("input_image_meta:0", shape=(?, ?), dtype=float32)
Found Tensor("output_detections:0", dtype=float32)
Found Tensor("output_mrcnn_class:0", shape=(?, 1000, 81), dtype=float32)
Found Tensor("output_mrcnn_bbox:0", shape=(?, 1000, 81, 4), dtype=float32)
Found Tensor("output_mrcnn_mask:0", shape=(?, 100, 28, 28, 81), dtype=float32)
Found Tensor("output_rois:0", shape=(1, ?, ?), dtype=float32)
Windows (1, 4) [[ 0 172 1024 852]]
2018-03-22 17:06:46.632334: W tensorflow/core/framework/op_kernel.cc:1192] Unknown: KeyError: 'pyfunc_0'
@fastlater Can you give me some advice? thank you .

[liangbo-1]

@ypflll Do you use mask_rcnn_coco.h5 to get .pb? i use the mask_rcnn_coco.h5 to get .pb and then,when i used @fastlater' script to test the .pb file, I had the following errors.
Windows (1, 4) [[ 0 172 1024 852]]
2018-03-22 19:48:29.487264: W tensorflow/core/framework/op_kernel.cc:1192] Unknown: KeyError: 'pyfunc_0'
Could you give me some advices?thank you

[ypflll]

@liangbo-1 No. I think you should train your own model before convert.

[liangbo-1]

@ypflll

Now I use my own dataset to train the model, convert it into .pb, and then use the script, but still have the same error. Is it related to the size of the picture? What size is the picture you use?

[ypflll]

@liangbo-1 I think your code is not the newest.

[ypflll]

@fastlater Hi, I am trying to convert your code to c++. It's a little complicated to call sess->run() 5 times to get the raw results and then post process to get the final results. Do you think the post-processing can be done inside the model so we can get the results(masks and rois coordinates) just call sess->run() one time?
Another question: I am not very clear about how to get the tensor by name. Like: detectionsT = sess.graph.get_tensor_by_name('output_detections:0'). Actually, in the model, there's no tensor named 'output_detections'. So what's the rule to get it?

[fastlater]

@ypflll Thank you for your feedback.

1. From the model.py

detections, mrcnn_class, mrcnn_bbox, mrcnn_mask, \
            rois, rpn_class, rpn_bbox =\
            self.keras_model.predict([molded_images, image_metas], verbose=0)

you can notice that the original code outputs each result separately. Since I haven't edit the original model, I had to use sess.run several times. If you know more about tensorflow and keras, maybe you can propose a way to "encapsulate/group/Concatenate" all the results into a single output tensor.

About the names of the outputs (for example, output_detections), it is the name defined in the exporting demo script. Feel free to follow my jupyter notebook demo where I put together both codes (exporting script prepared by @ericj974 and my importing script) link . This is the same code used before but I don't know why, I get error in sess = tf.session() in jupyter. I hav been trying to solve the problem so I changed that code line to tf.InteractiveSession() (based on some webpages) but still get error. If someone can make it run properly, let me know what do I need to change. At the end, this issue was created to share, discuss and help each other.

@githubXiaosong @liangbo-1 let me know if you found a way to solve that error. In my side, the script was running without problem and I cannot reproduce that error so I dont know what to say about it. I hope you can find the solution and share the solution with the rest of the community. Feel free to test the jupyter notebook mentioned in this comment too.

[ivshli]

@fastlater @ypflll
Hi, there is a code example about to convert code into C++ write by @moorage which you need only call it once and can get all outputs, I haven't test it yet, but your are very welcome to test and give the feedback about it, let's make the community more active :)
See code here:
https://github.com/matterport/Mask_RCNN/issues/222#issuecomment-373130661

[ypflll]

@fastlater Still not clear why tensor name like 'output_detections' not defined in the model but you can get it. I guess keras add a prefix 'output_' automatically here:
model = KM.Model([input_image, input_image_meta], [detections, mrcnn_class, mrcnn_bbox, mrcnn_mask, rpn_rois, rpn_class, rpn_bbox], name='mask_rcnn')

@ivshli I'll test it.

[moorage]

@ypflll if you look at the export to .pb code here: https://github.com/ericj974/Mask_RCNN/blob/master/scripts/export_model.py#L38

It prepends output_ to each node.

[ypflll]

@moorage That's exactly what I want!!

[samhodge]

OK I have been hacking away with @moorage 's source over here:
https://github.com/samhodge/OpenCVTensorflowExample/blob/master/OpenCVTensorflowExample

but putting in @moorage 's snippet from
https://github.com/matterport/Mask_RCNN/issues/222

it is currently commented out but when it does run

Currently it is crashing at:
https://github.com/samhodge/OpenCVTensorflowExample/blob/master/OpenCVTensorflowExample/main.cpp#L258

Clearly from the attached screenshot the dimensions are:

not

[1,100,28,28,2]

but
[1,100,28,28,81]

I assume the 81 refers to the number classifications, but

So how do we get from the 81 that we have to the 2 that @moorage's snippet is expecting?

from
https://github.com/matterport/Mask_RCNN/issues/222

Anyway my saved model is here:
https://drive.google.com/open?id=1xMqHddE14dWPi6PreyjqQg_hT_CNntsw

Has anyone else made progress getting this to run in C++?

Sam

[moorage]

@samhodge I only had 2 channels because I only trained two classes -- it's a good point that you brought that up. You'll have to change my code to expect the number of classes in your trained system.

In particular, these lines will have to change:
https://github.com/samhodge/OpenCVTensorflowExample/blob/master/OpenCVTensorflowExample/main.cpp#L277-L279

[samhodge]

Thanks for your time @moorage.

I will see what I can mix up

[ypflll]

@liangbo-1 I met this when I am using an old version code. Try to pull the newest code.

[liangbo-1]

@ypflll Thank you for your reply. Can you share your newest code? Thank you

[ypflll]

@liangbo-1 I mean the newest code of this repo.

[AliceDinh]

I have same error with @Surmeh.

Using TensorFlow backend.
Input file specified (mask_rcnn_mymodel.h5) only holds the weights, and not the model definition.
    Save the model using mode.save(filename.h5) which will contain the network architecture
    as well as its weights.

    If the model is saved using model.save_weights(filename.h5), the model architecture is
    expected to be saved separately in a json format and loaded prior to loading the weights.
    Check the keras documentation for more details (https://keras.io/getting-started/faq/)

Traceback (most recent call last):
  File "keras_to_tensorflow.py", line 123, in <module>
    raise err
  File "keras_to_tensorflow.py", line 114, in <module>
    net_model = load_model(weight_file_path)
  File "C:\Users\Alice\Anaconda3\lib\site-packages\keras\models.py", line 241, in load_model
    raise ValueError('No model found in config file.')
ValueError: No model found in config file.

So I reckon that, I have to modify my code at model.py by: mode.save(filename.h5) instead of model.save_weights(filename.h5), but I could see only this line relevant:

        callbacks = [
            keras.callbacks.TensorBoard(log_dir=self.log_dir,
                                        histogram_freq=0, write_graph=True, write_images=False),
            keras.callbacks.ModelCheckpoint(self.checkpoint_path,
                                            verbose=0, save_weights_only=True),
        ]

I should change: save_weights_only=False, right?
Pleas help

[ypflll]

@liangbo-1 Didn't meet this.
Maybe you can load and run your .h5 file before convert to .pb. Also you may check the inference_config.

[AliceDinh]

@Adriel-M and @Cpruce: Please please show me how did you fix the error: ValueError: Unknown layer: BatchNorm
Thanks

[AliceDinh]

@ericj974 : Please help me here. I run your script and here is the way I did it:
https://github.com/ericj974/Mask_RCNN/tree/master/scripts

[1] I copies codes inside the function def export() and put under the evaluation part.
[2] At the end of evaluation part, I put model.keras_model.save(filename) with filename = 'mask_rcnn.pb' hardcoded.
[3] I also provide the model_filepath = os.path.join(ROOT_DIR, "models/mask_rcnn_mymodel.h5").

And it is working like a charm, the output is mask_rcnn.pb but I reckoned that the size of this .pb file is exactly the same as original .h5 file. I really confused here that the process of converting is successfully or not.

[ypflll]

@liangbo-1 I use this:
https://github.com/ericj974/Mask_RCNN/blob/master/scripts/export_model.py

[liangbo-1]

@fastlater @ypflll I used your script: https://github.com/fastlater/Mask_RCNN/blob/master/demo_export_import_model.ipynb and got the error:
/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
/home/lb/fastlater-MASK-RCNN/Mask_RCNN-master/logs
/home/lb/fastlater-MASK-RCNN/Mask_RCNN-master/logs/ec20180321T0844/mask_rcnn_ec_0045.h5
2018-04-08 19:36:48.102086: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-04-08 19:36:48.196196: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-04-08 19:36:48.196454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 7.48GiB
2018-04-08 19:36:48.196468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
Model loaded.
['output_detections', 'output_mrcnn_class', 'output_mrcnn_bbox', 'output_mrcnn_mask', 'output_rois', 'output_rpn_class', 'output_rpn_bbox']
Converted 690 variables to const ops.
Saving frozen graph mask_rcnn_New.pb ...
3364 ops in the frozen graph.
Graph loaded.
2018-04-08 19:36:52.382194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
Image loaded.
Processing 1 images
image shape: (640, 1024, 3) min: 0.00000 max: 255.00000
RGB image loaded and preprocessed.
IMAGE_PADDING: True
(640, 1024, 3)
Image resized at: (1024, 1024, 3)
(192, 0, 832, 1024)
1
Image molded
Meta of image prepared
(1, 1024, 1024, 3)
Images meta: [[ 0 640 1024 3 192 0 832 1024 0 0 0]]
Tensor("input_image:0", shape=(?, 1024, 1024, 3), dtype=float32)
Tensor("input_image_meta:0", shape=(?, ?), dtype=float32)
Found Tensor("output_detections:0", dtype=float32)
Found Tensor("output_mrcnn_class:0", shape=(?, 1000, 3), dtype=float32)
Found Tensor("output_mrcnn_bbox:0", shape=(?, 1000, 3, 4), dtype=float32)
Found Tensor("output_mrcnn_mask:0", shape=(?, 100, 28, 28, 3), dtype=float32)
Found Tensor("output_rois:0", shape=(1, ?, ?), dtype=float32)
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1323, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
status, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value conv1/kernel
[[Node: conv1/kernel/read = Identity[T=DT_FLOAT, _class=["loc:@conv1/kernel"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv1/kernel)]]
[[Node: output_detections/_79 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4323_output_detections", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "infere_from_pb_new.py", line 315, in
detections = sess.run(detectionsT, feed_dict={img_ph: molded_images, img_meta_ph: image_metas})
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value conv1/kernel
[[Node: conv1/kernel/read = Identity[T=DT_FLOAT, _class=["loc:@conv1/kernel"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv1/kernel)]]
[[Node: output_detections/_79 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4323_output_detections", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'conv1/kernel/read', defined at:
File "infere_from_pb_new.py", line 90, in
model_dir=train_log_dirpath)
File "/home/lb/fastlater-MASK-RCNN/Mask_RCNN-master/model.py", line 1735, in __init__
self.keras_model = self.build(mode=mode, config=config)
File "/home/lb/fastlater-MASK-RCNN/Mask_RCNN-master/model.py", line 1791, in build
_, C2, C3, C4, C5 = resnet_graph(input_image, "resnet101", stage5=True)
File "/home/lb/fastlater-MASK-RCNN/Mask_RCNN-master/model.py", line 152, in resnet_graph
x = KL.Conv2D(64, (7, 7), strides=(2, 2), name='conv1', use_bias=True)(x)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 592, in __call__
self.build(input_shapes[0])
File "/usr/local/lib/python3.5/dist-packages/keras/layers/convolutional.py", line 138, in build
constraint=self.kernel_constraint)
File "/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(args, *kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 416, in add_weight
constraint=constraint)
File "/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py", line 395, in variable
v = tf.Variable(value, dtype=tf.as_dtype(dtype), name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py", line 213, in __init__
constraint=constraint)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py", line 356, in _init_from_args
self._snapshot = array_ops.identity(self._variable, name="read")
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/array_ops.py", line 125, in identity
return gen_array_ops.identity(input, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 2071, in identity
"Identity", input=input, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value conv1/kernel
[[Node: conv1/kernel/read = Identity[T=DT_FLOAT, _class=["loc:@conv1/kernel"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv1/kernel)]]
[[Node: output_detections/_79 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4323_output_detections", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Then I found that I need to add this sentence to your code:
sess = tf.Session()
sess.run(tf.initialize_all_variables()) and it won't get the same error. However, the final visualization results are wrong. Or it's disordered or unable to frame the location of the target object. My trained.H5 files work very well. @fastlater @ypflll Can you give me some advices? thank you

[fastlater]

@liangbo-1 If you are in hurry, try to use the notebook as a reference for your own script. I recommend you to run the script as a py file and not as jupyter notebook. I still dont know why it can run without error in python but outputs this kind of error using jupyter notebook.

[ypflll]

@liangbo-1 Maybe you can check you input again. Also, try different interpolation methods when resize.

[IgorKasianenko]

Hi, thanks for a lot of useful information!
Several questions:
@samhodge You shared a model.pb. What is your model dimensions? 2 or 81 classes? Does it contain person class?
I'm still confused, the model that you shared is keras model in protobuf, isn't it?

And main question to all, how did you overcome "Unknown layer: BatchNorm"?

@Cpruce could you confirm that you successfully convert model.h5 with keras_to_tensorflow.py?
If yes, may you be so kind to share result as samhodge did in his comment?

[samhodge]

The model is a frozen graph to run in Tensorflow it has 81 classes and includes person.

[samhodge]

@IgorKasianenko
I was testing out my .pb file I put on google drive and it doesnt seem to be detecting much according to https://github.com/samhodge/OpenCVTensorflowExample/commit/d5784b6787f1fdca3a78daf732bf77221573f5c8

But I might have made a dumb error like BGR rather than RGB, if someone want to play with the code there, you dont need XCode I can upload a Makefile for Linux if I get a chance. Basically you need libtensorflow_cc.so and libtensorflow_framework.so in your LD_LIBRARY_PATH and when you link.

[praneethkalimisetty]

can someone post the video how to open and run this repository to visualize the graph in tensorboard

https://github.com/ericj974/Mask_RCNN/blob/master/scripts/export_model.py
please give a video on how to open and view this file @ericj974
@Surmeh

[EscVM]

@fastlater Have you found a way to concatenate all outputs in a single output node?

[kuonangzhe]

@fastlater
Hi, I just ran your code. Everything works awesome in make .pb file, but I got a weird problem during pb file inference, telling that nms is not workable on GPU. I used your code with python files instead of notebook. The error details are as follows:

InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'ROI/rpn_non_max_suppression/NonMaxSuppressionV2_1': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
     [[Node: ROI/rpn_non_max_suppression/NonMaxSuppressionV2_1 = NonMaxSuppressionV2[_device="/device:GPU:0"](ROI/strided_slice_21_1, ROI/strided_slice_22_1, ROI/rpn_non_max_suppression/NonMaxSuppressionV2/max_output_size_1, ROI/rpn_non_max_suppression/iou_threshold_1)]]

Does this related to the version issues, or any other suggestions? I am using tf 1.7, keras 2.1.5, CUDA 8.0. Thank you so much!

There are also two other problems, but can be dealt with work around methods. One is for input_image_meta, when there are two classes (one is background), from compose_image_meta I get shape (1, 10), but in graph it is (?, 14). Another problem is that it asks for input_anchors, which needs to add to feed_dict in session run.

[parai]

Hi all:
with the https://github.com/ericj974/Mask_RCNN/blob/master/scripts/export_model.py, I sucessfully get the pb generated, but how to use it, is there any demo python script to load this pb and use it to process images and give some output just like the demo.ipynb.

PS:
I add below 3 line of code to the demo.ipynb to get the pb generated.

sys.path.append(os.path.join(ROOT_DIR, "mrcnn"))
from scripts.export_model import export
export(config, MODEL_DIR, COCO_MODEL_PATH)

2018-05-23 18:15:52.814432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-05-23 18:15:52.814478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2018-05-23 18:15:52.814797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4708 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
Loading weights from /media/data/Mask_RCNN/mask_rcnn_coco.h5
Converted 690 variables to const ops.
Saving frozen graph mask_rcnn.pb ...
3712 ops in the frozen graph.

[parai]

Hi all:
with some work and modification, I could run the pb now in the tensorflow.
check this commit https://github.com/parai/Mask_RCNN/commit/6289c1bd08fc90a1c3e296be8155674651f82a4b

[jezelmae12400]

@parai Thank you so much for this! I just got 1 problem

KeyError: "The name 'input_anchors_1:0' refers to a Tensor which does not exist. The operation, 'input_anchors_1', does not exist in the graph."

[gustavz]

@parai did you measure/time your inference pipeline?
how long does the processing of one image take (i mean the whole loop, not just the detection part)?
Furthermore You are creating a session for each output, why not combining them all into one single session with one single output dictionary?

[parai]

@GustavZ Currently I am a new begainer in the AI, I just try to bring the code of others up for the 2.0 release. And I didn't do any performance test.

[parai]

@jezelmae12400 are you sure you are using a pb that converted from the 2.0 release mask_rcnn_coco.h5

[parai]

Hi all:
I am trying to study the graph by the tensorboard, but I really can't understand the kind of shape such as shape=(?, ?, ?, 3), what it is, a 4D shape, is there any help document to introduce it, I can't find much valuable information on net.
Tensor("input_image_1:0", shape=(?, ?, ?, 3), dtype=float32)
Tensor("input_anchors_1:0", shape=(?, ?, 4), dtype=float32)
Tensor("input_image_meta_1:0", shape=(?, 93), dtype=float32)
Found Tensor("output_detections:0", shape=(1, 100, 6), dtype=float32)
Found Tensor("output_mrcnn_class:0", shape=(?, 1000, 81), dtype=float32)
Found Tensor("output_mrcnn_bbox:0", shape=(?, 1000, 81, 4), dtype=float32)
Found Tensor("output_mrcnn_mask:0", shape=(?, 100, 28, 28, 81), dtype=float32)
Found Tensor("output_rois:0", shape=(1, ?, ?), dtype=float32)
Windows (1, 4) [[ 128 0 896 1024]]

[chohaku84]

hi @AliceDinh and @ericj974
https://github.com/matterport/Mask_RCNN/issues/url
above link has expired but I really need help with how to save the model(not only the weights), could you repost the functions?

[ericj974]

@chohaku84
New script here

[chohaku84]

@ericj974 Thank you very much.

[xjohnxjohn]

@jmtatsch are you solved the problem coco.py save the model?

[jmtatsch]

@xjohnxjohn yes. I can post it on friday.

[VinniaKemala]

@jmtatsch How did you fix ValueError: Unknown layer: BatchNorm ?

[jmtatsch]

@xjohnxjohn @VinniaKemala try using this gist

[patrick-llgc]

@liangbo-1 If you are in hurry, try to use the notebook as a reference for your own script. I recommend you to run the script as a py file and not as jupyter notebook. I still dont know why it can run without error in python but outputs this kind of error using jupyter notebook.

@liangbo-1 @fastlater
Thanks for the notebook from @fastlater. It is very helpful. I figured out why there was an error in the jupyter notebook. You have to use tf.reset_default_graph() before loading the graph from pb again. Otherwise there seem to be multiple copies of the graph (as we built it from scratch earlier in the notebook), and tensorflow cannot match the tensors with the correct weights.

[slothkong]

@jmtatsch Quick question. Was it necessary to get rid or replace the custom 'BatchNorm' layers? I noticed you were also having problems with them.

[slothkong]

@fastlater You say that the reason why your code(https://github.com/fastlater/Mask_RCNN/blob/master/demo_export_import_model.ipynb) to evaluate images with the pb model yields an error is because is running inside a jupyter notebook? That is weird. I guess I will give it a try by moving the code a .py script.

[patrick-llgc]

@slothkong see my comment right above yours. The reason is that there are two networks in the default graph. In order to run it in the ipynb, you have to reset the default graph.

[fastlater]

@slothkong As @patrick-12sigma mentioned, I made a little mistake when I copied the code from the py script to the jupyter notebook. The reason I created the notebook was to illustrate more clearly the code execution since I knew the code was working fine in my side. Good to know it was a starting point for other notebooks.
Whenever I have time, I will try to add the fix.

@EscVM I haven't try to do that yet. did you make it?

[slothkong]

@patrick-12sigma @liangbo-1 @fastlater I can confirm that adding tf.reset_default_graph() right before the call to tf.gfile.FastGFile() makes it possible to load the graph from pb.

However, I noticed that the model has 3 inputs:

Tensor("input_image:0", shape=(?, ?, ?, 3), dtype=float32)
Tensor("input_image_meta:0", shape=(?, 93), dtype=float32)
Tensor("input_anchors:0", shape=(?, ?, 4), dtype=float32)

but code in @fastlater's notebook does not put any data in the anchor's tensor. When I run the session to get detections tensorflow realizes this and spits this error:

InvalidArgumentError: You must feed a value for placeholder tensor 'input_anchors' with dtype float and shape [?,?,4]
     [[Node: input_anchors = Placeholder[dtype=DT_FLOAT, shape=[?,?,4], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

You guys don't have the same issue?

EDIT: I just realized that @parai posted useful code in this script

[PreethaVeera]

@chohaku84
New script here

@ericj974
I am trying to convert the mask_rcnn_coco.h5 file into tensorflow .pb format. I tried using the keras to tensorflow code. But I am facing with the issue :
raise ValueError('No model found in config file.')
I tried your script still I am facing with the config file missing error. I have added the stack trace.
Using TensorFlow backend.

Traceback (most recent call last):
File "export.py", line 102, in
_main_(args)
File "export.py", line 34, in _main_
config_inference = load_config_files(config_path=config_path)
File "export.py", line 91, in load_config_files
assert os.path.exists(config_path)
File "/home/mcw/.conda/envs/py35/lib/python3.6/genericpath.py", line 19, in exists
os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

How to fix this?

[ericj974]

@PreethaVeera You need to
(1) Create a my_config.py containing a class MyConfig that will extend Config class in config.py
(1-a) add class property NAME = 'choose_a_name' (it will override the default value of None)
(1-b) add class property NUM_CLASSES = 81 (it will override the default value of 1)

(2) launch the export script where you specify
(2-a) model_path = path_to_h5
(2-b) log_dirpath = ''
(2-c) config_path = path_to_my_config_file

=> output pb file will be created in the same folder as your h5

[jbrownkramer]

@ericj974 Let me start by thanking you. Your script has created a pb file from my h5 file where no other script I used was able to.

However, I didn't exactly follow the comment above. Instead, I modified a copy of config.py to update NAME and NUM_CLASSES in the Config class itself and pointed the script to it. In particular, I tried putting the following code in its own file and pointing the script to it, and it didn't work.

#This doesn't work
import mrcnn.config as config

class MyConfig(config.Config):
    NAME = "MyConfig"
    NUM_CLASSES = 81

I get the following error

C:\Python36\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Traceback (most recent call last):
  File "export.py", line 102, in <module>
    _main_(args)
  File "export.py", line 34, in _main_
    config_inference = load_config_files(config_path=config_path)
  File "export.py", line 96, in load_config_files
    config = getattr(mod, Config.__name__)
AttributeError: module 'configs_' has no attribute 'Config'

Any insight into what I'm doing wrong?

[ifrogy]

@jbrownkramer Try:
from mrcnn.config import Config
class MyConfig(Config):

[javierfs]

I have the following issue:

this is root_dir: /home/javier/disk_user/OCR_project/Mask_RCNN/infering
this is Image_dir: /home/javier/disk_user/OCR_project/Mask_RCNN/infering/../customImages/val/
choose image file is 490_2.jpg
(5856, 4128, 3)
Processing 1 images
image                    shape: (5856, 4128, 3)       min:    0.00000  max:  255.00000  uint8
RGB image loaded and preprocessed.
/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/skimage/transform/_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images.
  warn("Anti-aliasing will be enabled by default in skimage 0.15 to "
(5856, 4128, 3)
Image resized at:  (1024, 1024, 3)
(0, 151, 1024, 873)
0.17486338797814208
Image molded
Meta of image prepared
(1, 1024, 1024, 3)
anchors shape is (1, 261888, 4) float32
Tensor("input_image:0", shape=(?, ?, ?, 3), dtype=float32, device=/device:CPU:0)
Tensor("input_anchors:0", shape=(?, ?, 4), dtype=float32, device=/device:CPU:0)
Tensor("input_image_meta:0", shape=(?, 14), dtype=float32, device=/device:CPU:0)
Found  Tensor("output_detections:0", shape=(2, 100, 6), dtype=float32)
Found  Tensor("output_mrcnn_class:0", shape=(?, 1000, 2), dtype=float32)
Found  Tensor("output_mrcnn_bbox:0", shape=(?, 1000, 2, 4), dtype=float32)
Found  Tensor("output_mrcnn_mask:0", shape=(2, 100, 28, 28, 2), dtype=float32)
Found  Tensor("output_rois:0", shape=(2, ?, ?), dtype=float32)
Windows (1, 4)   [[   0  151 1024  873]]
Traceback (most recent call last):
  File "infere_from_pb.py", line 318, in <module>
    tf.app.run()
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "infere_from_pb.py", line 78, in main
    img_anchors_ph: image_anchors})
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (1, 93) for Tensor 'input_image_meta:0', which has shape '(?, 14)'

Anyone knows why I got this?

[javierfs]

I have the following issue:

this is root_dir: /home/javier/disk_user/OCR_project/Mask_RCNN/infering
this is Image_dir: /home/javier/disk_user/OCR_project/Mask_RCNN/infering/../customImages/val/
choose image file is 490_2.jpg
(5856, 4128, 3)
Processing 1 images
image                    shape: (5856, 4128, 3)       min:    0.00000  max:  255.00000  uint8
RGB image loaded and preprocessed.
/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/skimage/transform/_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images.
  warn("Anti-aliasing will be enabled by default in skimage 0.15 to "
(5856, 4128, 3)
Image resized at:  (1024, 1024, 3)
(0, 151, 1024, 873)
0.17486338797814208
Image molded
Meta of image prepared
(1, 1024, 1024, 3)
anchors shape is (1, 261888, 4) float32
Tensor("input_image:0", shape=(?, ?, ?, 3), dtype=float32, device=/device:CPU:0)
Tensor("input_anchors:0", shape=(?, ?, 4), dtype=float32, device=/device:CPU:0)
Tensor("input_image_meta:0", shape=(?, 14), dtype=float32, device=/device:CPU:0)
Found  Tensor("output_detections:0", shape=(2, 100, 6), dtype=float32)
Found  Tensor("output_mrcnn_class:0", shape=(?, 1000, 2), dtype=float32)
Found  Tensor("output_mrcnn_bbox:0", shape=(?, 1000, 2, 4), dtype=float32)
Found  Tensor("output_mrcnn_mask:0", shape=(2, 100, 28, 28, 2), dtype=float32)
Found  Tensor("output_rois:0", shape=(2, ?, ?), dtype=float32)
Windows (1, 4)   [[   0  151 1024  873]]
Traceback (most recent call last):
  File "infere_from_pb.py", line 318, in <module>
    tf.app.run()
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "infere_from_pb.py", line 78, in main
    img_anchors_ph: image_anchors})
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (1, 93) for Tensor 'input_image_meta:0', which has shape '(?, 14)'

Anyone knows why I got this?

I solved that. I just have 2 classes. I had to change to inference_config.NUM_CLASSES = 2 from the script of @parai . Now I am getting a new error after running this line:

detections = sess.run(detectionsT, feed_dict={img_ph: molded_images, img_meta_ph: image_metas,
                                                      img_anchors_ph: image_anchors})

Error:

2018-12-20 15:05:52.008913: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 1 of dimension 0 out of bounds.
2018-12-20 15:05:52.009481: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 1 of dimension 0 out of bounds.
Traceback (most recent call last):
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 1 of dimension 0 out of bounds.
     [[{{node mrcnn_detection/strided_slice_34}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mrcnn_detection/truediv, mrcnn_detection/strided_slice_8/stack_1, mrcnn_detection/strided_slice_31/stack_1, mrcnn_detection/strided_slice_8/stack_1)]]
     [[{{node res5c_branch2c/bias/_426}} = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1078_res5c_branch2c/bias", _device="/job:localhost/replica:0/task:0/device:GPU:0"](res5c_branch2c/bias)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1664, in <module>
    main()
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/javier/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py", line 318, in <module>
    tf.app.run()
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py", line 78, in main
    img_anchors_ph: image_anchors})
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 1 of dimension 0 out of bounds.
     [[node mrcnn_detection/strided_slice_34 (defined at /home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py:29)  = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mrcnn_detection/truediv, mrcnn_detection/strided_slice_8/stack_1, mrcnn_detection/strided_slice_31/stack_1, mrcnn_detection/strided_slice_8/stack_1)]]
     [[{{node res5c_branch2c/bias/_426}} = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1078_res5c_branch2c/bias", _device="/job:localhost/replica:0/task:0/device:GPU:0"](res5c_branch2c/bias)]]

Caused by op 'mrcnn_detection/strided_slice_34', defined at:
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1664, in <module>
    main()
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/javier/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py", line 318, in <module>
    tf.app.run()
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py", line 29, in main
    _ = tf.import_graph_def(graph_def, name='')
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
    _ProcessNewOps(graph)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3440, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): slice index 1 of dimension 0 out of bounds.
     [[node mrcnn_detection/strided_slice_34 (defined at /home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py:29)  = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mrcnn_detection/truediv, mrcnn_detection/strided_slice_8/stack_1, mrcnn_detection/strided_slice_31/stack_1, mrcnn_detection/strided_slice_8/stack_1)]]
     [[{{node res5c_branch2c/bias/_426}} = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1078_res5c_branch2c/bias", _device="/job:localhost/replica:0/task:0/device:GPU:0"](res5c_branch2c/bias)]]

Anyone knows what would happen?

[ashnair1]

@parai I exported the model and then tried using the infere_from_pb.py script to run inference but ended up with this error:
KeyError: "The name 'input_image_1:0' refers to a Tensor which does not exist. The operation, 'input_image_1', does not exist in the graph."

Edit: Fixed it. My problem was a naming issue. Instead of input_image_1:0 it should have been input_image:0

Using @fastlater 's implementation I'm getting this error when running inference:
Cannot feed value of shape (1, 10) for Tensor 'input_image_meta:0', which has shape '(?, 14)'. This is the case using the infere_from_pb.py script and the notebook from @fastlater. I only have 2 classes (Background and building). it seems that some people have encountered this issue previously like @kuonangzhe and @javierfs , if so could you kindly tell me how you fixed it?

Edit: Got around it by reshaping the image_meta array.

On a side note has anyone been able to successfully go through the entire pipeline of exporting the model and using it for inference? There are so many variations here of exporters and importers which is a bit confusing.

[ashnair1]

So I was able to go through the entire pipeline of exporting and importing the model for inference. However the model isn't able to do inference correctly. I tried out the same model before and after exporting it to a .pb file on the same image to compare their results.

Standard Inference model
ROIs:
[[322 711 386 786]
[384 728 597 886]
[700 379 790 442]
[718 0 747 34]
[290 256 441 635]
[754 224 802 249]
[ 22 367 41 403]
[ 29 4 208 218]
[378 161 445 270]
[697 757 732 790]
[768 512 799 543]
[848 759 887 787]
[ 17 324 39 352]
[769 631 811 651]
[731 629 823 692]
[254 746 292 784]
[886 124 900 204]
[698 470 751 517]
[765 475 791 538]
[326 167 451 369]
[694 742 788 797]
[ 0 42 39 86]
[657 130 683 142]]

Imported .pb model:
ROIs:
[[1473 711 1474 786]
[1473 712 1474 783]
[1473 379 1474 442]
[1473 382 1474 443]
[1473 165 1474 276]
[1473 0 1474 34]
[1473 224 1474 249]
[1473 367 1474 403]
[1473 161 1474 270]
[1473 223 1474 247]
[1473 365 1474 400]
[1473 223 1474 247]
[1473 168 1474 281]
[1473 757 1474 790]
[1473 760 1474 792]
[1473 157 1474 285]
[1473 317 1474 649]
[1473 512 1474 543]
[1473 364 1474 399]
[1473 759 1474 787]
[1473 379 1474 449]
[1473 202 1474 669]
[1473 324 1474 352]
[1473 305 1474 682]
[1473 222 1474 247]
[1473 161 1474 278]
[1473 0 1474 34]
[1473 379 1474 440]
[1473 631 1474 651]
[1473 222 1474 248]
[1473 379 1474 445]
[1473 152 1474 399]
[1473 629 1474 692]
[1473 365 1474 400]
[1473 746 1474 784]
[1473 124 1474 204]
[1473 127 1474 210]
[1473 322 1474 351]
[1473 403 1474 447]
[1473 223 1474 249]
[1473 366 1474 401]
[1473 160 1474 312]
[1473 470 1474 517]
[1473 128 1474 213]
[1473 475 1474 538]
[1473 158 1474 262]
[1473 224 1474 251]
[1473 474 1474 513]
[1473 629 1474 648]
[1473 742 1474 797]
[1473 42 1474 86]
[1473 323 1474 350]
[1473 513 1474 544]
[1473 390 1474 448]
[1473 470 1474 520]
[1473 402 1474 447]
[1473 473 1474 514]
[1473 130 1474 142]
[1473 386 1474 445]
[1473 381 1474 444]]

So the model is,

1. Detecting more objects than are actually present
2. The first and third elements of the bounding box (x coordinate and width I believe) are stuck at a particular value (1473, 1474). This is also seen while testing on other images as well.

Any hints as to what could be the reason?

Edit: Never mind. Fixed it by rewriting the importer and exporter. Works fine now.

[a-ershov]

I received the .pb file, but I cannot use it for TensorFlow serving.
2019-02-12 12:39:17.963412: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve } 2019-02-12 12:39:17.975561: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:259] SavedModel load for tags { serve }; Status: fail. Took 99931 microseconds. 2019-02-12 12:39:17.975713: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name: r version: 1} failed: Not found: Could not find meta graph def matching supplied tags: { serve }. To inspect available tag-sets in the SavedModel, please use the SavedModel CLI: 'saved_model_cli'

And when I try to check using saved_model_cli I got error
RuntimeError: MetaGraphDef associated with tag-set could not be found in SavedModel

How to use the model with TF Serving?

[eoghanmull]

@chohaku84
New script here

What is the config file for --config_file=X ?

I had no config file for my training but set configurations through the config class.

[benDavidson6]

This will create a frozen graph which you can use

import keras.backend as K
import tensorflow as tf

# I needed to add this
sess = tf.Session()
K.set_session(sess)

from mrcnn import model as modellib
# my config subclass
from network_configs import ExampleConfig


def freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True):
    """
    Freezes the state of a session into a pruned computation graph.

    Creates a new computation graph where variable nodes are replaced by
    constants taking their current value in the session. The new graph will be
    pruned so subgraphs that are not necessary to compute the requested
    outputs are removed.
    @param session The TensorFlow session to be frozen.
    @param keep_var_names A list of variable names that should not be frozen,
                          or None to freeze all the variables in the graph.
    @param output_names Names of the relevant graph outputs.
    @param clear_devices Remove the device directives from the graph for better portability.
    @return The frozen graph definition.
    """
    graph = sess.graph

    with graph.as_default():
        freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or []))

        output_names = output_names or []
        input_graph_def = graph.as_graph_def()

        if clear_devices:
            for node in input_graph_def.node:
                node.device = ""

        frozen_graph = tf.graph_util.convert_variables_to_constants(
            session, input_graph_def, output_names, freeze_var_names)
        return frozen_graph


def freeze_model(model, name):
    frozen_graph = freeze_session(
        sess,
        output_names=[out.op.name for out in model.outputs][:4])
    directory = '.\\'
    tf.train.write_graph(frozen_graph, directory, name + '.pb', as_text=False)


config = ExampleConfig()
MODEL_DIR = 'where your weights live'
H5_WEIGHT_PATH = 'path to h5 file'
FROZEN_NAME = 'frozen_graph.pb
model = modellib.MaskRCNN(
    mode="inference",
    config=config,
    model_dir=MODEL_DIR)
model.load_weights(H5_WEIGHT_PATH, by_name=True)
freeze_model(model.keras_model, FROZEN_NAME)

[pknambiar]

@benDavidson6 Hello, Thanks for your code snippet. But, I didn't quite understand the first variable amongst these two below:

MODEL_DIR = 'where your weights live'
H5_WEIGHT_PATH = 'path to h5 file'

Is it that the former is the (input) PATH/DIR and the latter is the (input) .h5 file name ?

[benDavidson6]

ile'

Is it that the former is the (input)

The model dir is the folder your weights are in, and the path is the actual path to the weights.

[pknambiar]

Hi, The below line is not getting resolved in my code:
from network_configs import ExampleConfig
Where do I find network_configs ?

[benDavidson6]

That is my code. You need to adapt it so that you have a config object like config = ExampleConfig(). You must have created a config object somewhere to run any training or inference.

[davidb1]

Anyone knows how to get past the "Unknown layer: BatchNorm" error?

[marcosbis]

Anyone knows how to get past the "Unknown layer: BatchNorm" error?

Im having the same problem

[benDavidson6]

forward.zip

In the attached code there is a class Forward which you just give your config, and path to the pb file then you can do:

forward_model = Forward('path/to/pb/', my_config)
results = forward_model(list_of_image_arrays)

[changya1990]

forward.zip

In the attached code there is a class Forward which you just give your config, and path to the pb file then you can do:

forward_model = Forward('path/to/pb/', my_config)
results = forward_model(list_of_image_arrays)

Thanks, the result is right when the input node "input_image:0, input_image_meta:0" of my inference code changed to ''import/input_image:0, import/input_image_meta:0".

[navinthenapster]

@Surmeh Code here

This link is not available. Could you provide other link

[adebiasio-estilos]

How to use the model with TF Serving?

I have also this problem @Vellar , did you find any solution?

[bendangnuksung]

I converted the keras model to tensorflow frozen pb and tensorflow saved model. Code here.

[farmaker47]

This worked for me.
Pay attention to this line:
keras_to_tensorflow(model.keras_model,output_dir=output_dir,model_name="keras_segment.pb")
"where you have to write .....model.keras_model"

import os
import tensorflow as tf
import keras.backend as K

"Method for saving keras to tensorflow to a specific directory"
def keras_to_tensorflow(keras_model, output_dir, model_name,out_prefix="output_", 
log_tensorboard=True):

if os.path.exists(output_dir) == False:
    os.mkdir(output_dir)

out_nodes = []

for i in range(len(keras_model.outputs)):
    out_nodes.append(out_prefix + str(i + 1))
    tf.identity(keras_model.output[i], out_prefix + str(i + 1))

sess = K.get_session()

from tensorflow.python.framework import graph_util, graph_io

init_graph = sess.graph.as_graph_def()

main_graph = tf.compat.v1.graph_util.convert_variables_to_constants(sess, init_graph, out_nodes)

graph_io.write_graph(main_graph, output_dir, name=model_name, as_text=False)

if log_tensorboard:
    from tensorflow.python.tools import import_pb_to_tensorboard

    import_pb_to_tensorboard.import_to_tensorboard(
        os.path.join(output_dir, model_name),
        output_dir)

"Load weights"
 model.load_weights('mask_rcnn_coco.h5', by_name=True)

"Make directory"
output_dir = os.path.join(os.getcwd(),"checkpoint")

"Saving to Tensorflow"
keras_to_tensorflow(model.keras_model,output_dir=output_dir,model_name="keras_segment.pb")

print("Success!")

[frk1993]

I have the following issue:

this is root_dir: /home/javier/disk_user/OCR_project/Mask_RCNN/infering
this is Image_dir: /home/javier/disk_user/OCR_project/Mask_RCNN/infering/../customImages/val/
choose image file is 490_2.jpg
(5856, 4128, 3)
Processing 1 images
image                    shape: (5856, 4128, 3)       min:    0.00000  max:  255.00000  uint8
RGB image loaded and preprocessed.
/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/skimage/transform/_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images.
  warn("Anti-aliasing will be enabled by default in skimage 0.15 to "
(5856, 4128, 3)
Image resized at:  (1024, 1024, 3)
(0, 151, 1024, 873)
0.17486338797814208
Image molded
Meta of image prepared
(1, 1024, 1024, 3)
anchors shape is (1, 261888, 4) float32
Tensor("input_image:0", shape=(?, ?, ?, 3), dtype=float32, device=/device:CPU:0)
Tensor("input_anchors:0", shape=(?, ?, 4), dtype=float32, device=/device:CPU:0)
Tensor("input_image_meta:0", shape=(?, 14), dtype=float32, device=/device:CPU:0)
Found  Tensor("output_detections:0", shape=(2, 100, 6), dtype=float32)
Found  Tensor("output_mrcnn_class:0", shape=(?, 1000, 2), dtype=float32)
Found  Tensor("output_mrcnn_bbox:0", shape=(?, 1000, 2, 4), dtype=float32)
Found  Tensor("output_mrcnn_mask:0", shape=(2, 100, 28, 28, 2), dtype=float32)
Found  Tensor("output_rois:0", shape=(2, ?, ?), dtype=float32)
Windows (1, 4)   [[   0  151 1024  873]]
Traceback (most recent call last):
  File "infere_from_pb.py", line 318, in <module>
    tf.app.run()
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "infere_from_pb.py", line 78, in main
    img_anchors_ph: image_anchors})
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (1, 93) for Tensor 'input_image_meta:0', which has shape '(?, 14)'

Anyone knows why I got this?

I solved that. I just have 2 classes. I had to change to inference_config.NUM_CLASSES = 2 from the script of @parai . Now I am getting a new error after running this line:

detections = sess.run(detectionsT, feed_dict={img_ph: molded_images, img_meta_ph: image_metas,
                                                      img_anchors_ph: image_anchors})

Error:

2018-12-20 15:05:52.008913: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 1 of dimension 0 out of bounds.
2018-12-20 15:05:52.009481: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 1 of dimension 0 out of bounds.
Traceback (most recent call last):
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 1 of dimension 0 out of bounds.
   [[{{node mrcnn_detection/strided_slice_34}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mrcnn_detection/truediv, mrcnn_detection/strided_slice_8/stack_1, mrcnn_detection/strided_slice_31/stack_1, mrcnn_detection/strided_slice_8/stack_1)]]
   [[{{node res5c_branch2c/bias/_426}} = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1078_res5c_branch2c/bias", _device="/job:localhost/replica:0/task:0/device:GPU:0"](res5c_branch2c/bias)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1664, in <module>
    main()
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/javier/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py", line 318, in <module>
    tf.app.run()
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py", line 78, in main
    img_anchors_ph: image_anchors})
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 1 of dimension 0 out of bounds.
   [[node mrcnn_detection/strided_slice_34 (defined at /home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py:29)  = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mrcnn_detection/truediv, mrcnn_detection/strided_slice_8/stack_1, mrcnn_detection/strided_slice_31/stack_1, mrcnn_detection/strided_slice_8/stack_1)]]
   [[{{node res5c_branch2c/bias/_426}} = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1078_res5c_branch2c/bias", _device="/job:localhost/replica:0/task:0/device:GPU:0"](res5c_branch2c/bias)]]

Caused by op 'mrcnn_detection/strided_slice_34', defined at:
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1664, in <module>
    main()
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/javier/.pycharm_helpers/pydev/pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/javier/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py", line 318, in <module>
    tf.app.run()
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py", line 29, in main
    _ = tf.import_graph_def(graph_def, name='')
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
    _ProcessNewOps(graph)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3440, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/home/javier/anaconda3/envs/AIenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): slice index 1 of dimension 0 out of bounds.
   [[node mrcnn_detection/strided_slice_34 (defined at /home/javier/disk_user/OCR_project/Mask_RCNN/infering/infere_from_pb.py:29)  = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mrcnn_detection/truediv, mrcnn_detection/strided_slice_8/stack_1, mrcnn_detection/strided_slice_31/stack_1, mrcnn_detection/strided_slice_8/stack_1)]]
   [[{{node res5c_branch2c/bias/_426}} = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1078_res5c_branch2c/bias", _device="/job:localhost/replica:0/task:0/device:GPU:0"](res5c_branch2c/bias)]]

Anyone knows what would happen?

Hi,
I am facing the same error. Could you solve this problem ?

best regards

[buaacarzp]

@Surmeh Code here

I try to open the web from you give ,but it give me a 404 error ....

[buaacarzp]

forward.zip
In the attached code there is a class Forward which you just give your config, and path to the pb file then you can do:

forward_model = Forward('path/to/pb/', my_config)
results = forward_model(list_of_image_arrays)

Thanks, the result is right when the input node "input_image:0, input_image_meta:0" of my inference code changed to ''import/input_image:0, import/input_image_meta:0".

hey, I am the chinese too, my qq num is 510695983, I meet the same trouble with you ,but I didn't solve it , can you help me ?

[angyee]

@Surmeh Code here

please share code.. i have only .h5 then how to convert into .pb

[Surmeh]

Hi @angyee, the code was shared by @ericj974.

[angyee]

@jmtatsch
To produce h5:
python3 coco.py evaluate --dataset=$COCO_PATH --model=coco

To save model in coco.py:
evaluate_coco(model, dataset_val, coco, "bbox", limit=int(args.limit)) model.keras_model.save("mrcnn_eval.h5")

Extracting pb from h5:
python3 keras_to_tensorflow.py -input_model_file saved_model_mrcnn_eval.h5 -output_model_file model.pb -num_outputs=7

Could you paste the full stacktrace?

i am running

sudo python3 keras_to_tensorflow.py --input_model=/home/deepedge/mask_rcnn-master/mask_rcnn_damage_0010.h5 --output_model=/home/deepedge/mask_rcnn-master/dent.pb

and getting

Using TensorFlow backend.
E0619 15:22:44.379571 140399717234496 keras_to_tensorflow.py:95] Input file specified only holds the weights, and not the model definition. Save the model using model.save(filename.h5) which will contain the network architecture as well as its weights. If the model is saved using the model.save_weights(filename) function, either input_model_json or input_model_yaml flags should be set to to import the network architecture prior to loading the weights.
Check the keras documentation for more details (https://keras.io/getting-started/faq/)
Traceback (most recent call last):
File "keras_to_tensorflow.py", line 182, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "keras_to_tensorflow.py", line 128, in main
model = load_model(FLAGS.input_model, FLAGS.input_model_json, FLAGS.input_model_yaml)
File "keras_to_tensorflow.py", line 106, in load_model
raise wrong_file_err
File "keras_to_tensorflow.py", line 63, in load_model
model = keras.models.load_model(input_model_path)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py", line 221, in _deserialize_model
model_config = f['model_config']
File "/usr/local/lib/python3.6/dist-packages/keras/utils/io_utils.py", line 302, in getitem
raise ValueError('Cannot create group in read only mode.')
ValueError: Cannot create group in read only mode.

@Cpruce @Surmeh @waleedka @PavlosMelissinos

[ashish9277]

I am getting the same error as above. The error is occurring with original weights as well as with the custom Mask RCNN weights.

[lukeb94]

Could someone please inform me of the steps required to output a .h5 model file (not weights) that can then be converted to .pb. I understand how to convert to .pb but struggling with generating the model file. The above comments are somewhat helpful but some links are broken.

Help would be greatly appreciated! I have successfully setup a custom object detector with great results.

[angyee]

You can download the forward.zip provided in the comment and use the code below to inference it.

from forward import ForwardModel
import cv2
import numpy as np

frozen_graph_path = "frozen_model/mask_frozen_graph.pb"

assign your own Config class

my_config = YOUR_CONFIG_CLASS

forward_model = ForwardModel(frozen_graph_path, my_config)

def test_one_image(image):
images = np.expand_dims(image, axis=0)
results = forward_model(images)
return results

if __name__ == "__main__":
test_image_path = 'XYZ.jpg'
image = cv2.imread(test_image_path)
r = test_one_image(image)
print(r)

how to set my_config()?

@bendangnuksung @ashish9277 @gakarak

[lukeb94]

But how do we obtain either the .pb model file or the .h5 model file (not weight file) to then be converted into .pb? I am deploying the model on mobile using server.

[lukeb94]

Hi all, I successfully generated a .pb file with the help of #1126. Could someone please let me know how I can run inference on this .pb model file to test its functionality? Many thanks

[angyee]

Hi all, I successfully generated a .pb file with the help of #1126. Could someone please let me know how I can run inference on this .pb model file to test its functionality? Many thanks

I have also created the .pb by using now how to generate and display the masked image.
https://github.com/bendangnuksung/mrcnn_serving_ready

please solve same problem here please run the inference and share here
https://github.com/bendangnuksung/mrcnn_serving_ready/issues/5

@lukeb94 @bendangnuksung

[lukeb94]

Hi all, I successfully generated a .pb file with the help of #1126. Could someone please let me know how I can run inference on this .pb model file to test its functionality? Many thanks

I have also created the .pb by using now how to generate and display the masked image.
https://github.com/bendangnuksung/mrcnn_serving_ready

please solve same problem here please run the inference and share here
bendangnuksung/mrcnn_serving_ready#5

@lukeb94 @bendangnuksung

Yes, I managed to generate the .pb file. But how can we now test detentions?

Does this help? Very keen to get it working.
https://github.com/parai/Mask_RCNN/commit/6289c1bd08fc90a1c3e296be8155674651f82a4b

[ibrahimLearning]

@ericj974 @ps48 I created .pb file but how can I create .pbtxt file ot use in opencv

[ibrahimLearning]

hello all, what about .pbtxt file? would help me please. I created .pb but not .pbtxt. I need those files for opencv dnn
@ps48 @ericj974 @Surmeh @luoshanwei @lukeb94

[angyee]

hello all, what about .pbtxt file? would help me please. I created .pb but not .pbtxt. I need those files for opencv dnn
@ps48 @ericj974 @Surmeh @luoshanwei @lukeb94

https://gist.github.com/Arafatk/c063bddb9b8d17a037695d748db4f592

convert .pb to pbtxt
after that how to Display result please share me also

[ibrahimLearning]

@angyee
Thank you very much for your comment. I created .pb file using the following code:
import tensorflow as tf
from keras import backend as K
from tensorflow.python.framework import graph_util
import stone
from config import Config
import model as modellib
import os

def load_config_files():
config = stone.StoneConfig()
class InferenceConfig(config.__class__):
GPU_COUNT = 1
IMAGES_PER_GPU = 1
NAME="Broli"
NUM_CLASSES = 1 + 1
IMAGE_MIN_DIM = 512
IMAGE_MAX_DIM = 512
STEPS_PER_EPOCH = 100
VALIDATION_STEPS = 10
RPN_ANCHOR_SCALES = (16, 32, 64, 128, 256)
POST_NMS_ROIS_TRAINING = 2000
POST_NMS_ROIS_INFERENCE = 1000
config = InferenceConfig()
return config
DEVICE = "/gpu:0"
config_inference = load_config_files()
config_inference.display()
Model_Dir='logs/bro20190709T1941'
with tf.device(DEVICE):
model = modellib.MaskRCNN(mode="inference",
config=config_inference,
model_dir=Model_Dir)
model.load_weights('mask_rcnn.h5',by_name=True)
model_keras = model.keras_model
K.set_learning_phase(0)
num_output = 7
pred_node_names = ["detections", "mrcnn_class", "mrcnn_bbox", "mrcnn_mask", "rois", "rpn_class", "rpn_bbox"]
pred_node_names = ["output_" + name for name in pred_node_names]
pred = [tf.identity(model_keras.outputs[i], name=pred_node_names[i]) for i in range(num_output)]
sess = K.get_session()
od_graph_def = graph_util.convert_variables_to_constants(sess,sess.graph.as_graph_def(), pred_node_names)
filename = 'mrcnn_model.pb'
pb_filepath = os.path.join(Model_Dir, filename).replace("\","/")
print(pb_filepath)
print('Saving frozen graph {} ...'.format(os.path.basename(pb_filepath)))
frozen_graph_path = pb_filepath
with tf.gfile.GFile(frozen_graph_path, 'wb') as f:
f.write(od_graph_def.SerializeToString())

when I used the created .pb file in the opencv dnn,
net=cv2.dnn.readNetFromTensorflow('mrcnn_model.pb'
it gives the following error:
error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\dnn\src\tensorflow\tf_importer.cpp:497: error: (-2:Unspecified error) Input layer not found: ROI_7/split/value in function 'cv::dnn::dnn4_v20190122::anonymous-namespace'::TFImporter::connect'`

After creating .pbtxt file, I used opencv dnn,
net=cv2.dnn.readNetFromTensorflow('mrcnn_model.pb','frozen_graph.pbtxt')
but it gives the following error:
error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\dnn\src\tensorflow\tf_io.cpp:54: error: (-2:Unspecified error) FAILED: ReadProtoFromTextFile(param_file, param). Failed to parse GraphDef file: frozen_graph.pbtxt in function 'cv::dnn::ReadTFNetParamsFromTextFileOrDie'

What can I do? would you give some solution please?
@ps48 @ericj974 @Surmeh @Cpruce @dkurt

[angyee]

@angyee
Thank you very much for your comment. I created .pb file using the following code:
import tensorflow as tf
from keras import backend as K
from tensorflow.python.framework import graph_util
import stone
from config import Config
import model as modellib
import os

def load_config_files():
config = stone.StoneConfig()
class InferenceConfig(config.class):
GPU_COUNT = 1
IMAGES_PER_GPU = 1
NAME="Broccoli"
NUM_CLASSES = 1 + 1
IMAGE_MIN_DIM = 512
IMAGE_MAX_DIM = 512
STEPS_PER_EPOCH = 100
VALIDATION_STEPS = 10
RPN_ANCHOR_SCALES = (16, 32, 64, 128, 256)
POST_NMS_ROIS_TRAINING = 2000
POST_NMS_ROIS_INFERENCE = 1000
config = InferenceConfig()
return config
DEVICE = "/gpu:0"
config_inference = load_config_files()
config_inference.display()
Model_Dir='logs/broccoli20190709T1941'
with tf.device(DEVICE):
model = modellib.MaskRCNN(mode="inference",
config=config_inference,
model_dir=Model_Dir)
model.load_weights('mask_rcnn_broccoli.h5',by_name=True)
model_keras = model.keras_model
K.set_learning_phase(0)
num_output = 7
pred_node_names = ["detections", "mrcnn_class", "mrcnn_bbox", "mrcnn_mask", "rois", "rpn_class", "rpn_bbox"]
pred_node_names = ["output_" + name for name in pred_node_names]
pred = [tf.identity(model_keras.outputs[i], name=pred_node_names[i]) for i in range(num_output)]
sess = K.get_session()
od_graph_def = graph_util.convert_variables_to_constants(sess,sess.graph.as_graph_def(), pred_node_names)
filename = 'mrcnn_model.pb'
pb_filepath = os.path.join(Model_Dir, filename).replace("\","/")
print(pb_filepath)
print('Saving frozen graph {} ...'.format(os.path.basename(pb_filepath)))
frozen_graph_path = pb_filepath
with tf.gfile.GFile(frozen_graph_path, 'wb') as f:
f.write(od_graph_def.SerializeToString())

when I used the created .pb file in the opencv dnn,
net=cv2.dnn.readNetFromTensorflow('mrcnn_model.pb'
it gives the following error:
error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\dnn\src\tensorflow\tf_importer.cpp:497: error: (-2:Unspecified error) Input layer not found: ROI_7/split/value in function 'cv::dnn::dnn4_v20190122::anonymous-namespace'::TFImporter::connect'`

After creating .pbtxt file, I used opencv dnn,
net=cv2.dnn.readNetFromTensorflow('mrcnn_model.pb','frozen_graph.pbtxt')
but it gives the following error:
error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\dnn\src\tensorflow\tf_io.cpp:54: error: (-2:Unspecified error) FAILED: ReadProtoFromTextFile(param_file, param). Failed to parse GraphDef file: frozen_graph.pbtxt in function 'cv::dnn::ReadTFNetParamsFromTextFileOrDie'

What can I do? would you give some solution please?
@ps48 @ericj974 @Surmeh @Cpruce @dkurt

same problem here also.

[davidvirdeforshh]

I've trained a model, with great results, as follows:

model.load_weights('mask_rcnn_coco.h5', by_name=True, exclude=[
                       "mrcnn_class_logits", "mrcnn_bbox_fc",  "mrcnn_bbox", "mrcnn_mask"])
model.train(train_set, test_set, learning_rate=config.LEARNING_RATE, epochs=num_epochs, augmentation=augmentation, layers='heads')

It trains the last layer (heads) starting from mask_rcnn_coco.h5.

Now I want to export my model to a .pb-file. The goal is to create a Tensorflow Lite file to be used in a mobile application.

---

I've tried using 'keras_to_tensorflow.py'

https://github.com/amir-abdi/keras_to_tensorflow/blob/master/keras_to_tensorflow.py

But get the error

ValueError: Unknown layer: BatchNorm

I've tried freeze_model

https://github.com/matterport/Mask_RCNN/issues/1538#issuecomment-501932017

But get this error

Attempting to use uninitialized value bn4m_branch2a/moving_mean

I've set save_weights_only to False

save_weights_only=False

But got this error:
TypeError: can't pickle _thread.RLock objects

I've tried this freeze_model as described here:

https://github.com/matterport/Mask_RCNN/issues/1538#issuecomment-501932017

But got this error:
Attempting to use uninitialized value bn4m_branch2a/moving_mean

Anyone have a suggestion on how to proceed?

[bendangnuksung]

@davidvirdeforshh have you tried https://github.com/bendangnuksung/mrcnn_serving_ready ?
Edit: Direct link to the repo.

[davidvirdeforshh]

@davidvirdeforshh have you tried this

@bendangnuksung I tried it now and it produced a .pb file. Thank you, I really appreciate it!

[zambang83]

@all have successfully created the network graph.pb using https://github.com/bendangnuksung/mrcnn_serving_ready and created the text file.pbtxt of it using https://gist.github.com/Arafatk/c063bddb9b8d17a037695d748db4f592. I want to use my model in opencv to detect objects in the video when I load it the opencv dnn models for tensorflow, I get the following error message any solution for my challenge. Am using windows 10 corei7-6500U cpu of 2.50ghz quadcore a ram 20gb python 3.7.3 opencv version 4.1.0

net =cv2.dnn.readNetFromTensorflow( "mask_rcnn_0040_frozen_graph.pb", "mask_rcnn_0040.pbtxt")

error: (-2:Unspecified error) FAILED: ReadProtoFromTextFile(param_file, param). Failed to parse GraphDef file: mask_rcnn_0040.pbtxt in function 'cv::dnn::ReadTFNetParamsFromTextFileOrDie'

Thanks for your help

[kaulrohit]

@davidvirdeforshh did you managed to implement mask rcnn into mobile application?

[ahasan016]

@zambang83 Have you found any solution to your problem? I am facing the exact problem that you have mentioned.

[ArashHosseini]

Here also a workflow for h5 to pb https://github.com/matterport/Mask_RCNN/issues/1569#issuecomment-569908346

[bendangnuksung]

Added GRPC and RESTAPI client calls to tensorflow_model_server at https://github.com/bendangnuksung/mrcnn_serving_ready
Hope this helps someone.

[suchiz]

HI there, has anyone succeeded to export the model to Android ?

[davidvirdeforshh]

@davidvirdeforshh did you managed to implement mask rcnn into mobile application?

I gave up on mask rcnn for mobile.

[gospeljohn]

@davidvirdeforshh did you managed to implement mask rcnn into mobile application?

I gave up on mask rcnn for mobile.

@davidvirdeforshh Why gave up to migrate Android platform?

[ZhongOO]

Hi, I want to implement mask rcnn into mobile application. Now, I have converted suessfully into h5.
keras_file = "model.h5" keras.models.save_model(model.keras_model, keras_file)

But when I try to convert h5 to tflite, I get the error:
Traceback (most recent call last):
File "converth5.py", line 7, in
converter = lite.TFLiteConverter.from_keras_model_file(keras_file)
File "/home/hp/anaconda3/envs/yuki/lib/python3.6/site-packages/tensorflow/lite/python/lite.py", line 370, in from_keras_model_file
keras_model = _keras.models.load_model(model_file)
File "/home/hp/anaconda3/envs/yuki/lib/python3.6/site-packages/tensorflow/python/keras/engine/saving.py", line 232, in load_model
raise ValueError('No model found in config file.')
ValueError: No model found in config file.

What can I do to solve the problem?

[moganesyan]

Here's a repo of mine: https://github.com/moganesyan/tensorflow_model_deployment

It has code to export, serve and test the deployment locally (REST and GRPC). Jupyter notebooks provided

Hope it helps someone who is at the stage where I was a couple of months ago

[ZouJiu1]

convert-to pb file

#!encoding=utf-8
'''
#-----------------
Authors:邹九
Time:2019-11-21
#-----------------
'''
"""
Copyright (c) 2019, by the Authors: Amir H. Abdi
This script is freely available under the MIT Public License.
Please see the License file in the root for details.

The following code snippet will convert the keras model files
to the freezed .pb tensorflow weight file. The resultant TensorFlow model
holds both the model architecture and its associated weights.
"""

import tensorflow as tf
from tensorflow.python.framework import graph_util
from tensorflow.python.framework import graph_io
from pathlib import Path
from absl import app
from absl import flags
from absl import logging
from mrcnn import model as modellib
from mrcnn.config import Config
import keras
import os
from keras import backend as K
from keras.models import model_from_json, model_from_yaml
from keras.utils.vis_utils import plot_model

COCO_MODEL_PATH = r'../logs/shapes20191113T1540_mask_rcnn_shapes_0199.h5'

K.set_learning_phase(0)
FLAGS = flags.FLAGS

flags.DEFINE_string('input_model', default=r'', help='Path to the input model.')
flags.DEFINE_string('input_model_json', None, 'Path to the input model '
                                              'architecture in json format.')
flags.DEFINE_string('input_model_yaml', None, 'Path to the input model architecture in yaml format.')
flags.DEFINE_string('output_model', default=r'./shapes20191113T1540_mask_rcnn_shapes_0199.pb', help='Path where the converted model will be stored.')
flags.DEFINE_boolean('save_graph_def', False,
                     'Whether to save the graphdef.pbtxt file which contains '
                     'the graph definition in ASCII format.')
flags.DEFINE_string('output_nodes_prefix', None,
                    'If set, the output nodes will be renamed to '
                    '`output_nodes_prefix`+i, where `i` will numerate the '
                    'number of of output nodes of the network.')
flags.DEFINE_boolean('quantize', False,
                     'If set, the resultant TensorFlow graph weights will be '
                     'converted from float into eight-bit equivalents. See '
                     'documentation here: '
                     'https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/graph_transforms')
flags.DEFINE_boolean('channels_first', False,
                     'Whether channels are the first dimension of a tensor. '
                     'The default is TensorFlow behaviour where channels are '
                     'the last dimension.')
flags.DEFINE_boolean('output_meta_ckpt', False,
                     'If set to True, exports the model as .meta, .index, and '
                     '.data files, with a checkpoint file. These can be later '
                     'loaded in TensorFlow to continue training.')

flags.mark_flag_as_required('input_model')
flags.mark_flag_as_required('output_model')


def load_model(input_model_path, input_json_path=None, input_yaml_path=None):
    if not Path(input_model_path).exists():
        raise FileNotFoundError(
            'Model file `{}` does not exist.'.format(input_model_path))
    try:
        model = keras.models.load_model(input_model_path)
        return model
    except FileNotFoundError as err:
        logging.error('Input mode file (%s) does not exist.', FLAGS.input_model)
        raise err
    except ValueError as wrong_file_err:
        if input_json_path:
            if not Path(input_json_path).exists():
                raise FileNotFoundError(
                    'Model description json file `{}` does not exist.'.format(
                        input_json_path))
            try:
                model = model_from_json(open(str(input_json_path)).read())
                model.load_weights(input_model_path)
                return model
            except Exception as err:
                logging.error("Couldn't load model from json.")
                raise err
        elif input_yaml_path:
            if not Path(input_yaml_path).exists():
                raise FileNotFoundError(
                    'Model description yaml file `{}` does not exist.'.format(
                        input_yaml_path))
            try:
                model = model_from_yaml(open(str(input_yaml_path)).read())
                model.load_weights(input_model_path)
                return model
            except Exception as err:
                logging.error("Couldn't load model from yaml.")
                raise err
        else:
            logging.error(
                'Input file specified only holds the weights, and not '
                'the model definition. Save the model using '
                'model.save(filename.h5) which will contain the network '
                'architecture as well as its weights. '
                'If the model is saved using the '
                'model.save_weights(filename) function, either '
                'input_model_json or input_model_yaml flags should be set to '
                'to import the network architecture prior to loading the '
                'weights. \n'
                'Check the keras documentation for more details '
                '(https://keras.io/getting-started/faq/)')
            raise wrong_file_err

class ShapesConfig(Config):
    """Configuration for training on the toy shapes dataset.
    Derives from the base Config class and overrides values specific
    to the toy shapes dataset.
    """
    # Give the configuration a recognizable name
    NAME = "shapes"

    # Number of classes (including background)
    NUM_CLASSES = 1 + 14  # background + 15 object
    # Choose the number of GPU devices
    # os.environ['CUDA_VISIBLE_DEVICES'] = '0'

    # Use small images for faster training. Set the limits of the small side
    # the large side, and that determines the image shape.
    IMAGE_RESIZE_MODE = "square"
    IMAGE_MAX_DIM = 896

    RPN_ANCHOR_SCALES = (8 * 6, 16 * 6, 32 * 6, 64 * 6, 128 * 6)  # anchor side in pixels
    # RPN_ANCHOR_SCALES = (8*5, 16*5, 32*5, 64*5, 128*5)  # anchor side in pixels

    # Reduce training ROIs per image because the images are small and have
    # few objects. Aim to allow ROI sampling to pick 33% positive ROIs.
    TRAIN_ROIS_PER_IMAGE = 100

    # Use a small epoch since the data is simple
    # STEPS_PER_EPOCH = 1000
    STEPS_PER_EPOCH = 1000

    # use small validation steps since the epoch is small
    VALIDATION_STEPS = 25

def main(args):
    # If output_model path is relative and in cwd, make it absolute from root
    output_model = FLAGS.output_model
    if str(Path(output_model).parent) == '.':
        output_model = str((Path.cwd() / output_model))

    output_fld = Path(output_model).parent
    output_model_name = Path(output_model).name
    output_model_stem = Path(output_model).stem
    output_model_pbtxt_name = output_model_stem + '.pbtxt'

    # Create output directory if it does not exist
    Path(output_model).parent.mkdir(parents=True, exist_ok=True)

    if FLAGS.channels_first:
        K.set_image_data_format('channels_first')
    else:
        K.set_image_data_format('channels_last')

    # model = load_model(FLAGS.input_model, FLAGS.input_model_json, FLAGS.input_model_yaml)
    ##--------------------------------------------------------------------------------------#
    config = ShapesConfig()
    config.display()
    MODEL_DIR = r'E:\Desktop\Projects\Mask_RCNN-master\logs'
    model = modellib.MaskRCNN(mode="inference", config=config,\
                              model_dir=MODEL_DIR)
    model.load_weights(COCO_MODEL_PATH, by_name=True)#exclude=["mrcnn_class_logits", "mrcnn_bbox_fc",\
                                # "mrcnn_bbox", "mrcnn_mask"])
    # print(model.summary())
    # plot_model(model, to_file='model1.png', show_shapes=True)
    # model_json = model.to_json()
    # with open(r'./modle.json', 'w') as file:
    #     file.write(model_json)

    print('loaded model and saved json file')
    ##--------------------------------------------------------------------------------------#
    # TODO(amirabdi): Support networks with multiple inputs
    # orig_output_node_names = [node.op.name for node in model.outputs]
    orig_output_node_names = ['mrcnn_detection/Reshape_1', 'mrcnn_class/Softmax', 'mrcnn_bbox/Reshape',\
                              'mrcnn_mask/Sigmoid', 'ROI/packed_2', 'rpn_class/concat', 'rpn_bbox/concat']

    if FLAGS.output_nodes_prefix:
        num_output = len(orig_output_node_names)
        pred = [None] * num_output
        converted_output_node_names = [None] * num_output

        # Create dummy tf nodes to rename output
        for i in range(num_output):
            converted_output_node_names[i] = '{}{}'.format(
                FLAGS.output_nodes_prefix, i)
            pred[i] = tf.identity(model.outputs[i],
                                  name=converted_output_node_names[i])
    else:
        converted_output_node_names = orig_output_node_names
    logging.info('Converted output node names are: %s',
                 str(converted_output_node_names))

    sess = K.get_session()
    if FLAGS.output_meta_ckpt:
        saver = tf.train.Saver()
        saver.save(sess, str(output_fld / output_model_stem))

    if FLAGS.save_graph_def:
        tf.train.write_graph(sess.graph.as_graph_def(), str(output_fld),
                             output_model_pbtxt_name, as_text=True)
        logging.info('Saved the graph definition in ascii format at %s',
                     str(Path(output_fld) / output_model_pbtxt_name))

    if FLAGS.quantize:
        from tensorflow.tools.graph_transforms import TransformGraph
        transforms = ["quantize_weights", "quantize_nodes"]
        transformed_graph_def = TransformGraph(sess.graph.as_graph_def(), [],
                                               converted_output_node_names,
                                               transforms)
        constant_graph = graph_util.convert_variables_to_constants(
            sess,
            transformed_graph_def,
            converted_output_node_names)
    else:
        constant_graph = graph_util.convert_variables_to_constants(
            sess,
            sess.graph.as_graph_def(),
            converted_output_node_names)

    graph_io.write_graph(constant_graph, str(output_fld), output_model_name,
                         as_text=False)
    logging.info('Saved the freezed graph at %s',
                 str(Path(output_fld) / output_model_name))


if __name__ == "__main__":
    app.run(main)

load pb model

def load_detection_model(model):
    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    detection_graph = tf.Graph()
    with detection_graph.as_default():
        od_graph_def = tf.GraphDef()
        with tf.gfile.GFile(model, 'rb') as fid:
            serialized_graph = fid.read()
            od_graph_def.ParseFromString(serialized_graph)
            tf.import_graph_def(od_graph_def, name='')
        input_image = tf.get_default_graph().get_tensor_by_name('input_image:0')
        input_image_meta = tf.get_default_graph().get_tensor_by_name('input_image_meta:0')
        input_anchors = tf.get_default_graph().get_tensor_by_name('input_anchors:0')
        detections = tf.get_default_graph().get_tensor_by_name('mrcnn_detection/Reshape_1:0')
        mrcnn_mask = tf.get_default_graph().get_tensor_by_name('mrcnn_mask/Sigmoid:0')
    sessd=tf.Session(config=config,graph=detection_graph)
    print('Loaded detection model from file "%s"' % model)
    return sessd, input_image, input_image_meta, input_anchors, detections, mrcnn_mask

sessd, input_image, input_image_meta, input_anchors, detections, mrcnn_mask = load_detection_model(model_path)
results = model.detect_pb([image], sessd, input_image, input_image_meta, input_anchors, detections, mrcnn_mask,verbose=1)

use model, add to mrcnn/model.py

    def detect_pb(self, images, sessd, input_image, input_image_meta, input_anchors, detections, mrcnn_mask, verbose=1):
        """Runs the detection pipeline.

        images: List of images, potentially of different sizes.

        Returns a list of dicts, one dict per image. The dict contains:
        rois: [N, (y1, x1, y2, x2)] detection bounding boxes
        class_ids: [N] int class IDs
        scores: [N] float probability scores for the class IDs
        masks: [H, W, N] instance binary masks
        """
        assert self.mode == "inference", "Create model in inference mode."
        assert len(
            images) == self.config.BATCH_SIZE, "len(images) must be equal to BATCH_SIZE"

        # if verbose:
        #     log("Processing {} images".format(len(images)))
        #     for image in images:
        #         log("image", image)

        # Mold inputs to format expected by the neural network
        molded_images, image_metas, windows = self.mold_inputs(images)

        # Validate image sizes
        # All images in a batch MUST be of the same size
        image_shape = molded_images[0].shape
        # print(image_shape, molded_images.shape)
        for g in molded_images[1:]:
            assert g.shape == image_shape,\
                "After resizing, all images must have the same size. Check IMAGE_RESIZE_MODE and image sizes."

        # Anchors
        anchors = self.get_anchors(image_shape)
        # Duplicate across the batch dimension because Keras requires it
        # TODO: can this be optimized to avoid duplicating the anchors?
        anchors = np.broadcast_to(anchors, (self.config.BATCH_SIZE,) + anchors.shape)

        # if verbose:
        #     log("molded_images", molded_images)
        #     log("image_metas", image_metas)
        #     log("anchors", anchors)
        # Run object detection
        # detections, _, _, mrcnn_mask, _, _, _ =\
        #     self.keras_model.predict([molded_images, image_metas, anchors], verbose=0)
        detectionsed, mrcnn_masked = sessd.run([detections, mrcnn_mask], feed_dict = {input_image: molded_images, \
                                                               input_image_meta: image_metas, \
                                                               input_anchors: anchors})
        mrcnn_masked = np.expand_dims(mrcnn_masked, 0)
        detections = np.array(detectionsed)
        mrcnn_mask = np.array(mrcnn_masked)
        # Process detections
        results = []
        for i, image in enumerate(images):
            xi = detections[i]
            yi = mrcnn_mask[i]
            moldedi = molded_images[i]
            windowsi = windows[i]
            final_rois, final_class_ids, final_scores, final_masks =\
                self.unmold_detections(detections[i], mrcnn_mask[i],
                                       image.shape, molded_images[i].shape,
                                       windows[i])
            results.append({
                "rois": final_rois,
                "class_ids": final_class_ids,
                "scores": final_scores,
                "masks": final_masks,
            })
        return results

[mak27arr]

@Surmeh Code here

Update link pleas 404 error