Mask_rcnn: Save MaskRCNN full keras model

[oriolorra]

Hi everyone!

I am trying to save the full model (not only the weight).
I use the command:

_model.keras_model.save(path_file)_
And it does not give any error.

But, when I try to load the full model shows an error, i am using these comands:

_from keras.models import load_model_
_model_test = load_model(path_file)_

And it shows this error:

_~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\models.py in load_model(filepath, custom_objects, compile)
268 raise ValueError('No model found in config file.')
269 model_config = json.loads(model_config.decode('utf-8'))
--> 270 model = model_from_config(model_config, custom_objects=custom_objects)
271
272 # set weights

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\models.py in model_from_config(config, custom_objects)
345 'Maybe you meant to use '
346 'Sequential.from_config(config)?')
--> 347 return layer_module.deserialize(config, custom_objects=custom_objects)
348
349

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\layers__init__.py in deserialize(config, custom_objects)
53 module_objects=globs,
54 custom_objects=custom_objects,
---> 55 printable_module_name='layer')

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\utils\generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
142 return cls.from_config(config['config'],
143 custom_objects=dict(list(_GLOBAL_CUSTOM_OBJECTS.items()) +
--> 144 list(custom_objects.items())))
145 with CustomObjectScope(custom_objects):
146 return cls.from_config(config['config'])

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py in from_config(cls, config, custom_objects)
2523 # First, we create all layers and enqueue nodes to be processed
2524 for layer_data in config['layers']:
-> 2525 process_layer(layer_data)
2526 # Then we process nodes in order of layer depth.
2527 # Nodes that cannot yet be processed (if the inbound node

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py in process_layer(layer_data)
2509
2510 layer = deserialize_layer(layer_data,
-> 2511 custom_objects=custom_objects)
2512 created_layers[layer_name] = layer
2513

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\layers__init__.py in deserialize(config, custom_objects)
53 module_objects=globs,
54 custom_objects=custom_objects,
---> 55 printable_module_name='layer')

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\utils\generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
136 if cls is None:
137 raise ValueError('Unknown ' + printable_module_name +
--> 138 ': ' + class_name)
139 if hasattr(cls, 'from_config'):
140 custom_objects = custom_objects or {}

ValueError: Unknown layer: BatchNorm_

Has anyone tried to save a full model? Is it possible? has anyone a solution?

Thanks,

[aashish-0393]

Please help if someone has done it before.
@oriolorra did you find anything ? any progress?? please help

[oriolorra]

Hi @aashish-0393,

Yes, I have been able to save the full keras model. There is a webpage inside keras wiki that explains how to create a custom layer. All you have to do is to write some more methods inside the custom layers of MaskRCNN.

See you

[NorthLatitudeOne]

@oriolorra could you share your sample code?

[PhanDuc]

You can make some modifications in the model.py. There will be keras.callbacks.ModelCheckpoint, where you can choose to save weight or full model.

[oriolorra]

Hi @NorthLatitudeOne,

Sorry for the delay. Here you have the model.py file. With this, I was able to save the full model with Keras.

Best regards,
model.py.txt

[shivasj]

I was able to save the model using the following code:

import keras
# Save the Keras Model
keras.models.save_model(model.keras_model,"mask_rcnn_coco.hdf5")

# Save weights
model.keras_model.save_weights("mask_rcnn_coco.h5")

# Save model to JSON
model_json = model.keras_model.to_json()
with open("mask_rcnn_coco.json", "w") as json_file:
    json_file.write(model_json)

But when i try to load the saved model, i get the following error:

from mrcnn.model import BatchNorm
model2 = keras.models.load_model("mask_rcnn_coco.hdf5",custom_objects={"BatchNorm": BatchNorm})

~/anaconda/envs/bc3/lib/python3.6/site-packages/keras/layers/core.py in call(self, inputs, mask)
685 if has_arg(self.function, 'mask'):
686 arguments['mask'] = mask
--> 687 return self.function(inputs, **arguments)
688
689 def compute_mask(self, inputs, mask=None):
~/anaconda/envs/bc3/lib/python3.6/site-packages/keras/layers/core.py in (t)
855 # Reshape to [batch, anchors, 2]
856 rpn_class_logits = KL.Lambda(
--> 857 lambda t: tf.reshape(t, [tf.shape(t)[0], -1, 2]))(x)
858
859 # Softmax on last dimension of BG/FG.
NameError: name 'tf' is not defined

Any idea on how i can make "tf" available in the Lambda function?

[ibrahimLearning]

@oriolorra

Would you tell me please, where you changed in model.py? If I use your shared model.py file and use model.keras_model.save(dirPath), then is it OK?

[ibrahimLearning]

@oriolorra
Would you tell me please, where you changed in model.py? If I use your shared model.py file and use model.keras_model.save(dirPath), then is it OK?
@oriolorra would you share your config and inferenceconfig data please. Beacause when I am going to run your model, it gives some error: such as
'InferenceConfig' object has no attribute 'TRAIN_BN' Also for config.IMAGE_META_SIZE
I am waiting for your response. thanks

[ibrahimLearning]

@oriolorra
I run your model.py file to save full model. The code is here:
`
with tf.device(DEVICE):
model = modellib.MaskRCNN(mode="training", config=config,
model_dir=MODEL_DIR)

init_with = "coco"

if init_with == "imagenet":
model.load_weights(model.get_imagenet_weights(), by_name=True)
elif init_with == "coco":
print("Loading weights ", COCO_MODEL_PATH)
model.load_weights(COCO_MODEL_PATH, by_name=True) "
elif init_with == "last":
model.load_weights(model.find_last()[1], by_name=True)

model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE,
epochs=2,
layers="all")

model_path = os.path.join(MODEL_DIR, "mask_rcnn1.h5")
model_path2 = os.path.join(MODEL_DIR, "mask_rcnn2.h5")
model.keras_model.save_weights(model_path)
model.keras_model.save(model_path2)
`

but it gives the following error:

TypeError: can't pickle _thread.RLock objects

How can I save full model?

[denizsimsek06]

@ibrahimLearning
Look at this guide: https://keras.io/layers/writing-your-own-keras-layers/
You have to add "build", "compute_output_shape", and "get_config" methods to the custom layers implemented in model.py.

Then when loading I used

new_model = tf.keras.models.load_model('convert_1.h5', custom_objects={'ProposalLayer': modellib.ProposalLayer, #add dict entries like above for each custom layer})

[ibrahimLearning]

@oriolorra
I used your model.py to save full model. But when I am going to save using model.keras_model.save(model_path; it gives the following error:
TypeError: can't pickle _thread.RLock objects

Is there any suggestion please?

Thanks

[oriolorra]

@ibrahimLearning, I do not which problem is this one.

Sorry

[suchiz]

For anyone who comes here again ... I've done a work around. I was unable to save directly the model after training, like mostly everybody. So what I've done was to save weights only during the training and then during the evaluate or the splash, I load the inference model with my trained weights like usually and I just call a model.keras_model.to_json() and then save the json after the model.detect. Hope it will work for you too ...

[gabola24]

Hi @NorthLatitudeOne,

Sorry for the delay. Here you have the model.py file. With this, I was able to save the full model with Keras.

Best regards,
model.py.txt

Hi @oriolorra, thanks for your help. Your fixes in model.py it manage to solve the Batchnorm layer export. But now im looking the way to manage the other layers by the same way. Did you export the model without pass the custom objects " ProposalLayer,PyramidROIAlign,DetectionLayer" ?

[Suchit210Jadhav]

Hi @NorthLatitudeOne,
Sorry for the delay. Here you have the model.py file. With this, I was able to save the full model with Keras.
Best regards,
model.py.txt

Hi @oriolorra, thanks for your help. Your fixes in model.py it manage to solve the Batchnorm layer export. But now im looking the way to manage the other layers by the same way. Did you export the model without pass the custom objects " ProposalLayer,PyramidROIAlign,DetectionLayer" ?

Hey, I might have solution for this. I faced similar issue. Here, I've given some info on how to save the model.

[Craftsman381]

@ibrahimLearning
Look at this guide: https://keras.io/layers/writing-your-own-keras-layers/
You have to add "build", "compute_output_shape", and "get_config" methods to the custom layers implemented in model.py.

Then when loading I used

new_model = tf.keras.models.load_model('convert_1.h5', custom_objects={'ProposalLayer': modellib.ProposalLayer, #add dict entries like above for each custom layer})

I tried this and got this error
TypeError: __init__() missing 2 required positional arguments: 'proposal_count' and 'nms_threshold'

[Suchit210Jadhav]

hi, I have a problem, seems like no one work on it.

Mask-RCNN on video to remove background issues

I'm done with image segmentation on image/video, now I didn't find any clue to remove the background of this segmented video.
In matterport Repo ---->
( https://colab.research.google.com/github/zaidalyafeai/Notebooks/blob/master/Mask_RCNN.ipynb)
they use the following code to remove the segmented _image_ background------>

`def segment(image, r):
idx = r['scores'].argmax()
mask = r['masks'][:,:,idx]
mask = np.stack((mask,)_3, axis=-1) mask = mask.astype('uint8') bg = 255 - mask * 255 mask_img = image_mask
result = mask_img+ bg
return result

segmentation = segment(image, r)
plt.subplots(1, figsize=(16, 16))
plt.axis('off')
plt.imshow(np.concatenate([image, segmentation], axis = 1))`

How can I use the same for the video to remove the background?
https://colab.research.google.com/drive/1akpTRMrebgCCvFV_B-G9ADfhEaoPnUSY#scrollTo=md3eCFXQbGue

Because it works kind of same way --> it converts the Video to images frames. And then its Process images, in last step its Turn processed images to output videos.

So, does removing background frame by frame of a video not work?
Does it work on single image?

[Suchit210Jadhav]

@ibrahimLearning
Look at this guide: https://keras.io/layers/writing-your-own-keras-layers/
You have to add "build", "compute_output_shape", and "get_config" methods to the custom layers implemented in model.py.
Then when loading I used
new_model = tf.keras.models.load_model('convert_1.h5', custom_objects={'ProposalLayer': modellib.ProposalLayer, #add dict entries like above for each custom layer})

I tried this and got this error
TypeError: init() missing 2 required positional arguments: 'proposal_count' and 'nms_threshold'

It happens because those parameters are dynamically defined in model.py; I've explicitly declared those variables, have a look here.