Mask_rcnn: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape.

[cgd306948366]

E:\Programs\Python\lib\site-packages\tensorflow\python\ops\gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Epoch 1/1

When run the traning，the console shows the warning above. How to solve this problem?

[jlognn]

import warnings
warnings.filterwarnings('ignore')

[hemikak]

Check this out https://github.com/tflearn/tflearn/issues/460#issuecomment-407225809

[waleedka]

@cgd306948366 You can safely ignore this warning. It's a preemptive warning from TensorFlow when it cannot be certain of the size of the generated tensor.

[shen338]

@waleedka I also run into this. But it cannot be ignored because it will terminate my training. My input image size is 1920x1080 with a V100 GPU, 32G memory.
And also, loading COCO pretrained model to train is OK, but load my own checkpoint does not work.

[echoTomei]

@waleedka I also run into this. But it cannot be ignored because it will terminate my training. My input image size is 1920x1080 with a V100 GPU, 32G memory.
And also, loading COCO pretrained model to train is OK, but load my own checkpoint does not work.

Hi, I also meet this problem when training maskrcnn, Have you solved this problem?

[DoubleMuL]

@waleedka I also run into this. But it cannot be ignored because it will terminate my training. My input image size is 1920x1080 with a V100 GPU, 32G memory.
And also, loading COCO pretrained model to train is OK, but load my own checkpoint does not work.

Hi, I also meet this problem when training maskrcnn, Have you solved this problem?

I meet this problem when training maskrcnn, too. Do you know how to solve it?

[Nhiem]

I got the issue with training MaskRCNN especially I use nucleus.py for iris dataset but it shows the error, then it's stop running anyone could help really appreciate for your help

Note I already using Ignore warning but it still stop running

C:\Users\admin.conda\envs\tensorflownew\lib\site-packages\tensorflow_core\python\framework\indexed_slices.py:425: UserWarning: Converting sparse IndexedSlices to a dense
Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "

[thsdd]

I also got the error like Nhiem for the indexed_slices.py while trying to train the Mask R-CNN network from immersivelimit.com on my own dataset.
the reason maybe was the low memory of chrome. I got maybe 50 tabs open at the moment. For batch size =1 image per GPU it worked for 10 steps of the first epoch, then the kernel died. after changing to 20 images per GPU (i got a very bad GPU, regular one, not especially for ML tasks) it just didn't start and died at the beginning.

I changed to a fresh Firefox window (just 1 tab open) and it seems to work well (atm step 56 of 100, epoch 1 of 5).
I got 64 Gb of RAM, so maybe it is a chrome/browser issue.

I hope it helps.

(Win 10 x64)