Mask_rcnn: I can't continue balloon.py on 1/30 epoch

[bestyuan]

the code runing longtime in 1/30 epoch without traceback, could anyone tell me what problem.
`
2019-03-10 10:20:02.210479: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:01:00.0
totalMemory: 12.00GiB freeMemory: 9.93GiB
2019-03-10 10:20:02.210812: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN Xp, pci bus id: 0000:01:00.0, compute capability: 6.1)
Training network heads

Starting at epoch 0. LR=0.001

Checkpoint Path: D:\deeplearning_cai\objiect_recognize\mask_RCNN\Mask_RCNN-master\data_marcnn\logs\balloon20190310T1020\mask_rcnn_balloon_{epoch:04d}.h5
Selecting layers to train
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
D:\study_software\Anaconda\lib\site-packages\tensorflow\python\ops\gradients_impl.py:96: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Epoch 1/30
`

[bestyuan]

solved it ,set use_multiprocessing = False and workers=1 in model.py.

[zhouyuangan]

@bestyuan what's your tensorflow' version?

[tangchen2]

solved it ,set use_multiprocessing = False and workers=1 in model.py.

sry, whats ur tensorflow version? i still stop at epoch 1/30 without error and i have set use_multiprocess = False, workers = 1

[bestyuan]

@bestyuan what's your tensorflow' version?
tensorflow-gpu 1.4.0

[bestyuan]

solved it ,set use_multiprocessing = False and workers=1 in model.py.

sry, whats ur tensorflow version? i still stop at epoch 1/30 without error and i have set use_multiprocess = False, workers = 1

tensorflow-gpu 1.4.0

[FengJunJian]

solved it ,set use_multiprocessing = False and workers=1 in model.py.

thanks! I have solved it!

[imtinan39]

i have no gpu but facing the same problem. now what should i do?