Mask_rcnn: Error while running with Multiple GPU

[noamb3]

Hello,

I manage to run the package correctly with GPU_COUNT = 1 and any count of IMAGES_PER_GPU.
However, when I try to set GPU_COUNT >=1 and use multiple GPU's it produces the following error (it's long I'm attaching only the last part):

...
InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 600 values, but the requested shape has 1200
[[Node: tower_1_3/mask_rcnn/mrcnn_detection/Reshape_1 = ReshapeT=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Can you please assist me ?

Tensorflow version: 1.9.0

Many thanks!
Noam

[nfajardo]

Did you fixed?

[noamb3]

no

[zylo117]

same here

[colegleason]

Hi, I am also having this issue with multiple GPUs.

[Fight-hawk]

i have fixed this probelm, modified the line 820 in model.py
return tf.reshape(
detections_batch,
[self.config.BATCH_SIZE // self.config.GPU_COUNT, self.config.DETECTION_MAX_INSTANCES, 6])

[Fight-hawk]

but i found that it is slower than single gpu, if any one have same problem, please let me know

[chAwater]

As @Fight-hawk mentioned, this issue can be fixed by PR #1082 .

For @Fight-hawk 's question:

but i found that it is slower than single gpu, if any one have same problem, please let me know

Please check https://github.com/matterport/Mask_RCNN/issues/708#issuecomment-423711203 .

[nfajardo]

i have fixed this probelm, modified the line 820 in model.py
return tf.reshape(
detections_batch,
[self.config.BATCH_SIZE // self.config.GPU_COUNT, self.config.DETECTION_MAX_INSTANCES, 6])

Thanks It Worked!!!!!

[Fight-hawk]

As @Fight-hawk mentioned, this issue can be fixed by PR #1082 .

For @Fight-hawk 's question:

but i found that it is slower than single gpu, if any one have same problem, please let me know

Please check #708 (comment) .

thanks a lot@chAwater