Maskrcnn-benchmark: How to add X-101-64x4d-FPN_2x?

[antocapp]

Hello to everyone,
Where do I have to make changes in order to replicate the X-101-64x4d-FPN_2x architecture?

I have tried for the moment just to set NUM_GROUPS: 64 and WIDTH_PER_GROUP: 4 in RESNETS key in config file, but loading converted pretrained weights on from Detectron (so caffe2) leads to poor results.

Can you help me figure out the correct way to set the same architecture X-101-64x4d-FPN_2x used in Detectron?

[fmassa]

You should just change https://github.com/facebookresearch/maskrcnn-benchmark/blob/f25c6cff92d32d92abe8965d68401004e90c8bee/configs/e2e_faster_rcnn_X_101_32x8d_FPN_1x.yaml#L24-L25
and also the pre-trained model path, and it should work off-the-shelf.

If it doesn't work, I'd recommend having a look at the weights that were loaded (they are printed in the beginning of the log). They should show all weights in the model. If that's not the case, let me know

[antocapp]

Hi @fmassa, the problem is that I trained that network with Detectron repository on my own custom dataset, and now I want to ship all to pytorch using maskrcnn-benchmark repository; however the results are not the same.

Are there any other things that I may change? I have also tried to remove the preprocessing on input images in COCODemo class in predictor.py file.

EDIT: I removed the part where

image_list = to_image_list(image, self.cfg.DATALOADER.SIZE_DIVISIBILITY)

but doing inference with that image (batch size equal to 1, shape [3, 1176, 1140]) an error occurs:

File "/home/antonio/repositories/maskrcnn-benchmark/maskrcnn_benchmark/modeling/backbone/fpn.py", line 62, in forward
last_inner = inner_lateral + inner_top_down
RuntimeError: The size of tensor a (143) must match the size of tensor b (144) at non-singleton dimension 3

Could you please help on this step? A big thanks to you

[fmassa]

Hi,

You need to keep the size_divisibility if your model has FPN.

Here are a few things that I'd check:

• what did you change in the original Detectron model? Did you add extra layers? Did you change the name of any of the layers?
• Did you put the path to your pickled weights? Also can you verify that the individual weights are loaded properly? This is logged in the beginning of the log.

[antocapp]

Hi @fmassa, at the end I was able to reproduce the network. Thanks for the support.
What is the difference between using cfg.DATALOADER.SIZE_DIVISIBILITY or feeding the entire image to the network?
Thanks again

[fmassa]

What was the problem?

What is the difference between using cfg.DATALOADER.SIZE_DIVISIBILITY or feeding the entire image to the network?

Some models (like FPN) require that the image sizes are a multiple of a factor (because it performs upsampling with a fixed scale), that's why the SIZE_DIVISIBILITY is needed.

[antocapp]

I have changed the preprocessing operation performed in this line:
https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/demo/predictor.py#L195

so I created my own preprocessing, which is

def preprocess(self, im):
        """
        Creates a basic transformation that was used to train the models
        """
        cfg = self.cfg

        target_size = 800
        max_size = 1333

        pixel_means=cfg.INPUT.PIXEL_MEAN
        im = im.astype(np.float32, copy=False)
        im -= pixel_means
        im_shape = im.shape
        im_size_min = np.min(im_shape[0:2])
        im_size_max = np.max(im_shape[0:2])
        im_scale = float(target_size) / float(im_size_min)
        # Prevent the biggest axis from being more than max_size
        if np.round(im_scale * im_size_max) > max_size:
            im_scale = float(max_size) / float(im_size_max)
        im = cv2.resize(
            im,
            None,
            None,
            fx=im_scale,
            fy=im_scale,
            interpolation=cv2.INTER_LINEAR
        )
        return torch.tensor(im.transpose(2,0,1))

which is basically the preprocessing used in detectron here: https://github.com/facebookresearch/Detectron/blob/master/detectron/utils/blob.py#L100-L125

[antocapp]

Moreover, to overcome the previous error:

File "/home/antonio/repositories/maskrcnn-benchmark/maskrcnn_benchmark/modeling/backbone/fpn.py", line 62, in forward
last_inner = inner_lateral + inner_top_down
RuntimeError: The size of tensor a (143) must match the size of tensor b (144) at non-singleton dimension 3

I changed the line in the fpn backbone https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/modeling/backbone/fpn.py#L57-L61 by removing the comment on inner_top_down at line 60 and commenting line 57

[fmassa]

@antocapp the error you faced is also fixed (following Detectron implementation) by considering SIZE_DIVISIBILITY

[antocapp]

Moreover, to overcome the previous error:

File "/home/antonio/repositories/maskrcnn-benchmark/maskrcnn_benchmark/modeling/backbone/fpn.py", line 62, in forward
last_inner = inner_lateral + inner_top_down
RuntimeError: The size of tensor a (143) must match the size of tensor b (144) at non-singleton dimension 3

I changed the line in the fpn backbone https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/modeling/backbone/fpn.py#L57-L61 by removing the comment on inner_top_down at line 60 and commenting line 57

Never mind about this because I changed again the configuration as it is in the repo and it works now

[fmassa]

About the bad accuracy that you had, it was probably due to a problem with the image size, or the BGR image order, as the implementation that we have should be almost equivalent to what you added.
Anyway, if it's working for you now it's great!

[antocapp]

I will try to use the build_transform() function again and let you know what is the problem. Do you agree that I should set --min-image-size to be 800 as it is in Detectron target_size?

[antocapp]

Hi @fmassa, I have tried again now with build_transforms and everything is just fine. Sorry to have bothered you but maybe the mistake was just to not have set --min-image-size to 800...

[fmassa]

Cool, good to know, thanks!