Mask_rcnn: Failed on TF 1.8

[bertocast]

Hi folks,

I've just update Tensorflow to version 1.8.
A script that I've been using without errors, now it fails with:

F ./tensorflow/core/util/cuda_launch_config.h:127] Check failed: work_element_count > 0 (0 vs. 0)

I had to downgrade TF back to version 1.6 to get it ok again.

[jameschartouni]

I've found TF 1.5 to be the most stable. Avoid using TF 1.8 until the library has been updated.

[snarb]

Tf 1.7 works, 1.8 - not.

[patrickcgray]

I was nearly maxing out the memory of my machine training a network and got this error. Downgrading to 1.7.0 also worked for me.

[waleedka]

It seems that 1.8 might be less stable. It worked for me, though, on Ubuntu 18. I'll share my installation details below in case it helps.

Unlike previous installations, where I followed the long instructions on the Tensorflow website, this time I was wondering if I can do everything with conda. After all, PyTorch makes it really easy to install, so why can't TF? So I did a bit of research and experiments and and it worked. These lines installed everything needed for this project.

conda install cudatoolkit cudnn tensorflow-gpu keras cython opencv
pip install imgaug
pip install git+https://github.com/waleedka/coco.git#subdirectory=PythonAPI

[patrickcgray]

@waleedka that all worked for me as well on Ubuntu 16.04 until I was nearly maxing out memory with my trials to increase mask size. Those same easy commands but adding tensorflow-gpu=1.7.0 kept it simple but is not throwing the occasional errors that tensorflow-gpu=1.8.0 had. I love how easy that is to set up though! And conda has saved me a number of times when I've experimented with some weird setups.