Mask_rcnn: Training balloon model on gpu: AttributeError: 'Model' object has no attribute 'metrics_tensors'

Created on 19 Sep 2019 · 38Comments · Source: matterport/Mask_RCNN

I am trying to train a model with the balloon example. I can get the training running on my local cpu, but when I attempt to run the balloon.py on my gpu (and actually also from the Jupyter Notebook) I get the error below. Everything seemed to install fine, bot requirements and the setup.py.

python balloon.py train dataset=/home/Mask_RCNN/samples/balloon/balloon/ --weights=mask_rcnn_coco.h5

Traceback (most recent call last): File "/home/bitcue/Dropbox/mrcnn2/Mask_RCNN/samples/balloon/balloon.py", line 367, in <module> train(model) File "/home/bitcue/Dropbox/mrcnn2/Mask_RCNN/samples/balloon/balloon.py", line 201, in train layers='heads') File "/home/bitcue/anaconda3/envs/mrcnn2/lib/python3.6/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py", line 2354, in train File "/home/bitcue/anaconda3/envs/mrcnn2/lib/python3.6/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py", line 2199, in compile AttributeError: 'Model' object has no attribute 'metrics_tensors

I have the following packages installed in an anaconda environment:

Name Version Build Channel
_libgcc_mutex 0.1 main
_tflow_select 2.1.0 gpu
absl-py 0.7.1 py36_0
astor 0.8.0 py36_0
backports 1.0 py_2
backports.weakref 1.0rc1 py36_0
blas 1.0 mkl
bleach 1.5.0 py36_0
c-ares 1.15.0 h7b6447c_1001
ca-certificates 2019.5.15 1 anaconda
certifi 2019.6.16 py36_1 anaconda
cudatoolkit 9.0 h13b8566_0
cudnn 7.6.0 cuda9.0_0
cupti 9.0.176 0
expat 2.2.6 he6710b0_0
gast 0.2.2 py36_0
git 2.20.1 pl526hacde149_0
grpcio 1.16.1 py36hf8bcb03_1
h5py 2.9.0 py36h7918eee_0
hdf5 1.10.4 hb1b8bf9_0
html5lib 0.9999999 py36_0
intel-openmp 2019.5 281
keras-applications 1.0.8 py_0
keras-gpu 2.0.8 py36h0585f72_0 anaconda
keras-preprocessing 1.1.0 py_1
krb5 1.16.1 h173b8e3_7
libcurl 7.65.3 h20c2e04_0
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc 7.2.0 h69d50b8_2
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libprotobuf 3.8.0 hd408876_0
libssh2 1.8.2 h1ba5d50_0
libstdcxx-ng 9.1.0 hdf63c60_0
markdown 3.1.1 py36_0
mkl 2019.5 281
mkl-service 2.3.0 py36he904b0f_0
mkl_fft 1.0.14 py36ha843d7b_0
mkl_random 1.0.2 py36hd81dba3_0
mock 3.0.5 py36_0
ncurses 6.1 he6710b0_1
numpy 1.16.5 py36h7e9f1db_0
numpy-base 1.16.5 py36hde5b4d6_0
openssl 1.1.1 h7b6447c_0 anaconda
perl 5.26.2 h14c3975_0
pip 19.2.2 py36_0
protobuf 3.8.0 py36he6710b0_0
python 3.6.9 h265db76_0
pyyaml 5.1.2 py36h7b6447c_0
readline 7.0 h7b6447c_5
scipy 1.3.1 py36h7c811a0_0
setuptools 41.0.1 py36_0
six 1.12.0 py36_0
sqlite 3.29.0 h7b6447c_0
tensorboard 1.13.1 py36hf484d3e_0
tensorflow 1.13.1 gpu_py36h26cf82e_0
tensorflow-base 1.13.1 gpu_py36h8f37b9b_0
tensorflow-estimator 1.13.0 py_0
tensorflow-gpu 1.13.1 h0d30ee6_0
tensorflow-gpu-base 1.7.0 py36hcdda91b_1
tensorflow-tensorboard 1.5.1 py36hf484d3e_1
termcolor 1.1.0 py36_1
tk 8.6.8 hbc83047_0
werkzeug 0.15.5 py_0
wheel 0.33.4 py36_0
xz 5.2.4 h14c3975_4
yaml 0.1.7 had09818_2
zlib 1.2.11 h7b6447c_3

Source

HjalteMann

👍6 👀3

Most helpful comment

If anyone is looking for the solution, downgrade keras to 2.1.0. It started working for me.

Hunain-Gillani on 20 Sep 2019

👍38 😄5 🚀2

All 38 comments

i meet the same question,How to solve it?

yangdeqieyingli on 20 Sep 2019

Initialise self.keras_model.metrics_tensors = [] then you are good to go...

msrkp on 20 Sep 2019

👍10

If anyone is looking for the solution, downgrade keras to 2.1.0. It started working for me.

Hunain-Gillani on 20 Sep 2019

👍38 😄5 🚀2

Thank you for the reply, @51r1u5 !
Where in the scripts should I do this?

@Hunain-G, I'm using keras 2.0.8 as advised in the readme. Do you propose to upgrade?

HjalteMann on 20 Sep 2019

@HjalteMann , if you're using keras 2.0.8 then there shouldn't have been any issue. I was using keras 2.3.0 and by downgrading keras to 2.1.0 made the issue resolved.

Hunain-Gillani on 21 Sep 2019

👍7

since Keras 2.3.0 there is such issue, when i downgrade to 2.2.5 the issue resolved so i believe any version <= 2.2.5 would work, and of course should be >= 2.0.8 as requiremets.txt suggested.

mashuai191 on 28 Sep 2019

👍5

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)

mffigueroa on 1 Oct 2019

👍32 ❤3

I got the same error but when trying to run train_shapes (as described here). Fixed by installing keras version 2.2.5 and tensorflow version 1.15.0rc2 (pip3 install keras==2.2.5 and the same formatting for TF). Keras version 2.1.0 did not work for me, nor did version 2.3.0.

JepsonNomad on 3 Oct 2019

👍9 🚀1 🎉1

Did someone succeed to run any example with tensorflow 1.13 ? This is the oldest version which is compatible with cuda10.0. Older versions require older cuda's, but installing several cuda's is "highly" inconvenient

Stancoo on 9 Oct 2019

I was having the same error, but after installing keras 2.0.8 as you said above. Now I have other error:
File "/Users/julianvidal/anaconda3/envs/MaskRCNN/lib/python3.6/site-packages/keras/callbacks/callbacks.py", line 23, in <module> from ..engine.training_utils import standardize_input_data ModuleNotFoundError: No module named 'keras.engine.training_utils'

Can you help please? Thanks!

ijulian1997 on 10 Oct 2019

Initialise self.keras_model.metrics_tensors = [] then you are good to go...

Can you give more details about how to do this? Thanks!

ijulian1997 on 10 Oct 2019

since Keras 2.3.0 there is such issue, when i downgrade to 2.2.5 the issue resolved so i believe any version <= 2.2.5 would work, and of course should be >= 2.0.8 as requiremets.txt suggested.

ran tf.keras.__version__ in the console in a Kaggle kernel and got '2.2.4-tf'. I'm struggling with the same issue, though, and I'm not sure how to downgrade on Kaggle.

rteuwens on 15 Oct 2019

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)

It's solve the problem for 2.3.0

trialzuki on 16 Oct 2019

👍1

My issue was solved by installing all dependencies in a new conda environment, with pip, not conda.

HjalteMann on 17 Oct 2019

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)

it is work! 非常感谢

dfy888 on 25 Dec 2019

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)

it is work! 非常感谢

hi excuse me @dfy888 how to edit model.py inside egg file? i can open it by rename the .egg into .zip and modify all file inside that egg, but unfortunately i don't know package it back as a
python egg file after my modification. thank you

pipun-gif on 28 Dec 2019

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)

it is work! 非常感谢

我应该在哪里修改这一句呢？

ZOUYIyi on 3 Jan 2020

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)
it is work! 非常感谢

我应该在哪里修改这一句呢？

around here : anaconda3/envs/mrcnn2/lib/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py

pipun-gif on 5 Jan 2020

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)
it is work! 非常感谢

我应该在哪里修改这一句呢？

around here : anaconda3/envs/mrcnn2/lib/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py

hi , first change model.py file's line 2199 , which is "self.keras_model.metrics_tensors.append(loss)" to "self.keras_model.add_metric(loss, name)" and then with this command "python .\setup.py install" generate new egg file. You cannot directly change the content of the egg file.

Also , I do not recommend downgrading tensorflow because in return , it generates performance issuses.

oktemerbay on 5 Jan 2020

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)
it is work! 非常感谢

我应该在哪里修改这一句呢？

mrcnn\model.py
line2199

tourist-C on 20 Jan 2020

i have keras version 2.2.0 and tf 1.16.0 python 3.6.10 and having this issue.... many said it will work on this... anyone has its fix? I tried multiple combinations of version till now.. nothing worked. thanks!

abhishekpundir22 on 30 Mar 2020

i have keras version 2.2.0 and tf 1.16.0 python 3.6.10 and having this issue.... many said it will work on this... anyone has its fix? I tried multiple combinations of version till now.. nothing worked. thanks!

Hey Abhishek, check out @mffigueroa 's answer:

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)

As @tourist-C points out, it's on line 2199 in mcrnn/model.py
As here: https://github.com/rteuwens/Mask_RCNN/blob/master/mrcnn/model.py#L2199

rteuwens on 30 Mar 2020

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)
it is work! 非常感谢

我应该在哪里修改这一句呢？

around here : anaconda3/envs/mrcnn2/lib/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py

This path doesn't exist for me. How can I find where the model.py is or where my mrcnn package is saved in anaconda?

banafsh89 on 16 Apr 2020

In my case, this works.

cd Mask_RCNN # Root directory of Mask RCNN
emacs mrcnn/model.py # Edit model.py

--------------------------------------

At line 2199
# self.keras_model.metrics_tensors.append(loss) # <- comment out
self.keras_model.add_metric(loss, name) # <- Add

--------------------------------------

sudo python3 setup.py install # <- mask_rcnn-2.1-py3.6.egg is automaticalliy replaced

Then, balloon training script will work.

Independent-Laboratory on 17 Apr 2020

👍5 ❤2 👀1 🚀1 🎉1

If anyone is looking for the solution, downgrade keras to 2.1.0. It started working for me.

It did not work for me, I got another error about keras.

Mstfakts on 3 May 2020

If anyone is looking for the solution, downgrade keras to 2.1.0. It started working for me.

It did not work for me, I got another error about keras.

Can you specify the error? What the error is now?

Hunain-Gillani on 3 May 2020

If anyone is looking for the solution, downgrade keras to 2.1.0. It started working for me.

It did not work for me, I got another error about keras.

Can you specify the error? What the error is now?

Actually, I do not exactly remember the error but @mffigueroa 's solution worked well.

Mstfakts on 3 May 2020

Initialise self.keras_model.metrics_tensors = [] then you are good to go...
well,I use keras version=2.3.1,it is not work for me.

gethubwy on 4 Jul 2020

I have solved this issue by changing keras version to 2.1.0 and tensorflow version to 1.15.0

abhishekpundir22 on 4 Jul 2020

i have keras version 2.2.0 and tf 1.16.0 python 3.6.10 and having this issue.... many said it will work on this... anyone has its fix? I tried multiple combinations of version till now.. nothing worked. thanks!

Hey Abhishek, check out @mffigueroa 's answer:

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)

As @tourist-C points out, it's on line 2199 in mcrnn/model.py
As here: https://github.com/rteuwens/Mask_RCNN/blob/master/mrcnn/model.py#L2199

This works too. I can confirm. Thanks.

abhishekpundir22 on 4 Jul 2020

i have keras version 2.2.0 and tf 1.16.0 python 3.6.10 and having this issue.... many said it will work on this... anyone has its fix? I tried multiple combinations of version till now.. nothing worked. thanks!

Hey Abhishek, check out @mffigueroa 's answer:

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)

As @tourist-C points out, it's on line 2199 in mcrnn/model.py
As here: https://github.com/rteuwens/Mask_RCNN/blob/master/mrcnn/model.py#L2199

This works too. I can confirm. Thanks.

"name“ is what?

gethubwy on 4 Jul 2020

Maybe its a parameter having name of the Metric function.. I am not fully sure as this was used in earlier versions. Above solution is a just a tweak. Can you give me your keras version and tensorflow version details?

abhishekpundir22 on 4 Jul 2020

In my case, this works.

cd Mask_RCNN # Root directory of Mask RCNN
emacs mrcnn/model.py # Edit model.py

--------------------------------------

At line 2199

self.keras_model.metrics_tensors.append(loss) # <- comment out

self.keras_model.add_metric(loss, name) # <- Add

--------------------------------------

sudo python3 setup.py install # <- mask_rcnn-2.1-py3.6.egg is automaticalliy replaced

Then, balloon training script will work.

Hello,bro, I wonder how do you distinguish between a instances_train2017.json file and instances_val2017.json file.
I have labled data file, I want to train a new model starting from pre-trained COCO weights, but I don't know how to name the json file,and the vaild json file should like what format? Thank you.

gethubwy on 4 Jul 2020

What about changing: self.keras_model.metrics_tensors.append(loss)
to: self.keras_model.add_metric(loss, name)
Hello,I want to train a new model starting from pre-trained COCO weights, but I don't know how to name the json file,and the vaild json file should like what format? Thank you.
When you train a new model ,I wonder how do you distinguish between a instances_train2017.json file and instances_val2017.json file.
I have labled data file,

gethubwy on 4 Jul 2020

As per the version I used keras 2.1.0 and for the format I recommend you to use VGG annotation tool. It produces the exact format that is required to feed the model. Its available online.

abhishekpundir22 on 4 Jul 2020

Maybe its a parameter having name of the Metric function.. I am not fully sure as this was used in earlier versions. Above solution is a just a tweak. Can you give me your keras version and tensorflow version details?

Excuse me, I use the software named "labelme" to label my data, I want to train a new model starting from pre-trained COCO weights, but I don't know how to write the annotation file, are there tools to help me? Thank you.

gethubwy on 5 Jul 2020

In my case, this works.

cd Mask_RCNN # Root directory of Mask RCNN
emacs mrcnn/model.py # Edit model.py

--------------------------------------

At line 2199

self.keras_model.metrics_tensors.append(loss) # <- comment out

self.keras_model.add_metric(loss, name) # <- Add

--------------------------------------

sudo python3 setup.py install # <- mask_rcnn-2.1-py3.6.egg is automaticalliy replaced

Then, balloon training script will work.
Excuse me, I want to know how to train a new mode, I use the software named "labelme" to label my data, I want to train a new model starting from pre-trained COCO weights, but I don't know how to write the annotation file, are there tools to help me? Thank you.

gethubwy on 5 Jul 2020

In my case, this works.

cd Mask_RCNN # Root directory of Mask RCNN
emacs mrcnn/model.py # Edit model.py

--------------------------------------

At line 2199

self.keras_model.metrics_tensors.append(loss) # <- comment out

self.keras_model.add_metric(loss, name) # <- Add

--------------------------------------

sudo python3 setup.py install # <- mask_rcnn-2.1-py3.6.egg is automaticalliy replaced

Then, balloon training script will work.
Excuse me, I want to know how to train a new mode, I use the software named "labelme" to label my data, I want to train a new model starting from pre-trained COCO weights, but I don't know how to write the annotation file, are there tools to help me? Thank you.