Yolact: What will happen if I change the backbone network into MobileNet-v2 with FPN?

[Wilbur529]

Hi,
I want to try to change the backbone network into MobileNet-v2 with FPN. Is there any suggestions? THX!!!

[dbolya]

That would be a good test, though I will warn that we've tried VGG16+FPN before and got sub-par performance (though maybe that's to be expected). If you want to change out the backbone head over to backbone.py and check out how I implemented Resnet and VGG. If you need any help, let me know.

[Wilbur529]

@dbolya Hi, I have done a preliminary test on my own simple dataset. And the experiment result made me feel excited~! With the help of MobileNet-v2, I successfully reduced the model size from over 100 MB to 31 MB, but without a significant performance loss on my own dataset. Next, i will try it on MS-COCO dataset, and i hope for the fantastic performance^_^

[dbolya]

Sounds good! If it ends up working well, I might just make an official MobileNet-v2 version for YOLACTv1.1. That seems like it would be nice to have.

[Wilbur529]

@dbolya Hi, could you show me your train log file of resnet&coco?

[Wilbur529]

This is my recently train log with mobilenetv2&coco...

[ 16] 59810 || B: 2.489 | C: 3.295 | M: 2.861 | S: 0.897 | T: 9.542 || ETA: 19 days, 23:21:26 || timer: 2.437

[ 16] 59820 || B: 2.505 | C: 3.321 | M: 2.840 | S: 0.898 | T: 9.565 || ETA: 19 days, 22:54:13 || timer: 2.125
[ 16] 59830 || B: 2.500 | C: 3.299 | M: 2.802 | S: 0.910 | T: 9.511 || ETA: 19 days, 22:34:59 || timer: 1.966
[ 16] 59840 || B: 2.504 | C: 3.271 | M: 2.787 | S: 0.900 | T: 9.461 || ETA: 19 days, 22:51:36 || timer: 2.071
[ 16] 59850 || B: 2.480 | C: 3.282 | M: 2.751 | S: 0.881 | T: 9.394 || ETA: 19 days, 22:43:14 || timer: 2.415
[ 16] 59860 || B: 2.462 | C: 3.247 | M: 2.752 | S: 0.883 | T: 9.344 || ETA: 19 days, 22:27:50 || timer: 2.144
[ 16] 59870 || B: 2.460 | C: 3.247 | M: 2.726 | S: 0.881 | T: 9.315 || ETA: 19 days, 22:22:21 || timer: 2.091
[ 16] 59880 || B: 2.489 | C: 3.290 | M: 2.696 | S: 0.891 | T: 9.366 || ETA: 19 days, 22:38:05 || timer: 2.101
[ 16] 59890 || B: 2.478 | C: 3.300 | M: 2.712 | S: 0.892 | T: 9.382 || ETA: 19 days, 22:48:17 || timer: 2.301
[ 16] 59900 || B: 2.485 | C: 3.299 | M: 2.711 | S: 0.889 | T: 9.384 || ETA: 19 days, 22:53:21 || timer: 2.046
[ 16] 59910 || B: 2.514 | C: 3.316 | M: 2.724 | S: 0.880 | T: 9.435 || ETA: 19 days, 23:27:14 || timer: 1.973
[ 16] 59920 || B: 2.505 | C: 3.308 | M: 2.721 | S: 0.877 | T: 9.410 || ETA: 19 days, 23:25:19 || timer: 2.257
[ 16] 59930 || B: 2.496 | C: 3.290 | M: 2.710 | S: 0.863 | T: 9.360 || ETA: 19 days, 23:15:29 || timer: 2.061
[ 16] 59940 || B: 2.467 | C: 3.269 | M: 2.694 | S: 0.864 | T: 9.293 || ETA: 19 days, 22:23:27 || timer: 1.924
[ 16] 59950 || B: 2.473 | C: 3.249 | M: 2.702 | S: 0.868 | T: 9.292 || ETA: 19 days, 22:55:27 || timer: 2.374
[ 16] 59960 || B: 2.497 | C: 3.261 | M: 2.702 | S: 0.858 | T: 9.318 || ETA: 19 days, 23:18:36 || timer: 2.055
[ 16] 59970 || B: 2.532 | C: 3.274 | M: 2.726 | S: 0.862 | T: 9.393 || ETA: 19 days, 23:51:10 || timer: 2.111
[ 16] 59980 || B: 2.493 | C: 3.262 | M: 2.723 | S: 0.861 | T: 9.339 || ETA: 19 days, 23:32:02 || timer: 1.991
[ 16] 59990 || B: 2.500 | C: 3.253 | M: 2.718 | S: 0.860 | T: 9.331 || ETA: 19 days, 23:44:44 || timer: 3.396

The loss value looks like going down. But the convergence speed looks so slowly. And i got a poor evaluated result:

box | 0.78 | 2.17 | 1.76 | 1.39 | 1.08 | 0.75 | 0.43 | 0.19 | 0.07 | 0.01 | 0.00 |
box | 1.31 | 3.44 | 2.92 | 2.37 | 1.78 | 1.31 | 0.78 | 0.33 | 0.14 | 0.04 | 0.00 |
box | 1.47 | 3.88 | 3.23 | 2.54 | 2.05 | 1.47 | 0.82 | 0.44 | 0.21 | 0.04 | 0.00 |
box | 2.17 | 5.26 | 4.63 | 3.84 | 3.12 | 2.26 | 1.42 | 0.77 | 0.27 | 0.11 | 0.00 |
box | 2.41 | 5.79 | 5.10 | 4.26 | 3.46 | 2.54 | 1.74 | 0.82 | 0.34 | 0.06 | 0.00 |
box | 2.65 | 6.06 | 5.34 | 4.67 | 3.76 | 2.90 | 1.96 | 1.05 | 0.51 | 0.20 | 0.03 |
box | 2.82 | 6.62 | 5.94 | 4.82 | 4.05 | 2.94 | 2.04 | 1.17 | 0.53 | 0.12 | 0.00 |

mask | 0.80 | 1.91 | 1.61 | 1.35 | 1.09 | 0.83 | 0.58 | 0.38 | 0.21 | 0.07 | 0.00 |
mask | 1.44 | 3.13 | 2.71 | 2.28 | 1.95 | 1.62 | 1.20 | 0.81 | 0.55 | 0.12 | 0.01 |
mask | 1.73 | 3.73 | 3.28 | 2.81 | 2.35 | 1.88 | 1.43 | 0.96 | 0.57 | 0.26 | 0.03 |
mask | 2.38 | 4.86 | 4.39 | 3.80 | 3.27 | 2.65 | 2.09 | 1.49 | 0.95 | 0.29 | 0.03 |
mask | 2.76 | 5.46 | 5.04 | 4.48 | 3.84 | 3.15 | 2.43 | 1.70 | 0.96 | 0.44 | 0.06 |
mask | 3.01 | 5.81 | 5.34 | 4.82 | 4.16 | 3.53 | 2.72 | 1.97 | 1.10 | 0.56 | 0.07 |
mask | 3.31 | 6.41 | 5.85 | 5.27 | 4.56 | 3.94 | 2.99 | 2.13 | 1.31 | 0.58 | 0.06 |

Hope to discuss more with you ^_^

[dbolya]

Hmm with the mAP starting that low (< 1), it looks likes the behavior we get when we train from scratch. Are you using imagenet pretrained weights in your MobileNet implementation? If not, it'll take way longer to converge.

There's also a gotcha when loading pretrained weights because I had to set strict=False in the torch.load for loading backbone weights. Set that back to True if you are using pretrained weights but suspect it's not loading properly.

[Wilbur529]

THX~!
I used the implementation and the pretrained weights from this repo:https://github.com/tonylins/pytorch-mobilenet-v2

I did the experiment follow these steps bellow:

1. Modify the network architecture by removing the final global average pooling layer and the final 1x1 conv layer;
2. Make a custom init_backbone() method in which i translated the node name and discarded the unuseful tensor;
3. Add a new config-dict to the 'config.py', and set some helpful parameters.

Here is my MobileNet-v2 implementation:

class MobileNetV2(nn.Module):
    def __init__(self):
        super(MobileNetV2, self).__init__()
        block = InvertedResidual
        input_channel = 32
        interverted_residual_setting = [
            # t, c, n, s
            [1, 16, 1, 1],
            [6, 24, 2, 2], # useful
            [6, 32, 3, 2], # useful
            [6, 64, 4, 2],
            [6, 96, 3, 1], # useful
            [6, 160, 3, 2],
            [6, 320, 1, 1], # useful
        ]

        # Select the output layers
        # idx-1 => 56 * 56 * 24
        # idx-2 => 28 * 28 * 32
        # idx-4 => 14 * 14 * 96
        # idx-6 => 7 * 7 * 320
        select_idxs = [1, 2, 4, 6]


        # building first layer
        input_channel = int(input_channel)
        self.conv2d = conv_bn(3, input_channel, 2)
        self.layers = nn.ModuleList()
        self.channels = []

        # building inverted residual blocks
        features = []
        for idx, (t, c, n, s) in enumerate(interverted_residual_setting):
            output_channel = int(c)

            for i in range(n):
                if i == 0:
                    features.append(block(input_channel, output_channel, s, expand_ratio=t))
                else:
                    features.append(block(input_channel, output_channel, 1, expand_ratio=t))
                input_channel = output_channel

            if idx in select_idxs:
                self.layers.append(nn.Sequential(*features))
                self.channels.append(output_channel)
                features = []

        self.backbone_modules = [m for m in self.modules() if isinstance(m, nn.Conv2d)]

        # random initial
        self._initialize_weights()

    def forward(self, x):
        x = self.conv2d(x)

        outs = []
        for layer in self.layers:
            x = layer(x)
            outs.append(x)

        return tuple(outs)

    def _initialize_weights(self):
        modules = self.modules()
        for m in modules:
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                if m.bias is not None:
                    m.bias.data.zero_()
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()

    def init_backbone(self, path):
        """ Initializes the backbone weights for training. """
        state_dict = torch.load(path, map_location='cpu')

        # Replace featuresXXX -> layers.x.xx(/conv2d.x) etc.
        keys = list(state_dict)
        for key in keys:
            if key.startswith('features'):
                idx = int(key.split('.')[1])
                if idx <= 17:
                    if idx == 0:
                        new_key = 'conv2d.' + key[11:]
                    elif (idx >= 1 and idx <= 3):
                        new_key = "layers.0." + str(idx - 1) + key[10:]
                    elif (idx >= 4 and idx <=6):
                        new_key = "layers.1." + str(idx - 4) + key[10:]
                    elif (idx >= 7 and idx <= 13):
                        if idx <= 9:
                            new_key = "layers.2." + str(idx - 7) + key[10:]
                        else:
                            new_key = "layers.2." + str(idx - 7) + key[11:]
                    else:
                        new_key = "layers.3." + str(idx - 14) + key[11:]

                    state_dict[new_key] = state_dict.pop(key)
                else:
                    state_dict.pop(key)
            else:
                state_dict.pop(key)


        # Note: Using strict=False is berry scary. Triple check this.nn.ModuleList()
        self.load_state_dict(state_dict, strict=False)

And there is the corresponding config:

mobilenetv2_backbone = resnet50_backbone.copy({
    'name': 'MobileNetV2',
    'path': 'xxx.pth',
    'type': MobileNetV2,
    'args': (),
    # 'transform': resnet_transform,

    'selected_layers': [1, 2, 3],
    'pred_scales': [[24], [48], [96], [192], [384]],
    'pred_aspect_ratios': [[[1.414]]] * 5,
    'use_pixel_scales': False,
})
yolact_mobilenetv2_coco_config = yolact_resnet50_config.copy({
    'name': 'yolact_mobilenetv2_coco',

    'backbone': mobilenetv2_backbone.copy(),

    'use_semantic_segmentation_loss': True
})

[Wilbur529]

haha, i found the problem...Just after i commented...
When i do test on my own dataset, i removed some other anchors on purpose... and i forgot to recover them.

[dbolya]

Hmm, can you still do a sanity check and change self.load_state_dict(state_dict, strict=False) to self.load_state_dict(state_dict, strict=True) and see if there's anything unexpected is missing? I think even with one anchor per layer, you shouldn't get < 1 mAP on the first validation epoch.

[Wilbur529]

Thx, i'm just going to do it!

[andeyeluguo]

So,what's the accuracy and fps with mobilenet v2 now?@Wilbur529

[Wilbur529]

@andeyeluguo
It has run almost 200 epochs now, and the mAP are 14.36(box) and 15.02(mask) respectively. But I found that the model does not converge completely, so keep waiting...

[abhigoku10]

@Wilbur529 wonderful work on the modification on the network backbone . keep going !!!!!!!!!!

[abhigoku10]

@Wilbur529 hi is it possible to share your mobilenet implementation py ?? how good is the accuracy and fps compared to resenet 50,101

[Wilbur529]

@abhigoku10 You could find the model and the accuracy in my previous answers, and i will do a benchmark test later. As far as I can see from the training log, i think it would be vary fast.

[abhigoku10]

@Wilbur529 i was able to train the model using mobilenet architecture, thanks for sharing the reference implementation , when i tested it on images its able to detect with low accuracy but when i run it for a video the detection are not happening at all . Did u face the similar issue , should we make any changes with the code

[Wilbur529]

@abhigoku10 Good job! I think it may not be a difficult problem. If you couldn't find the error code, why not write a for-loop upon your image inference function to simulate the video inference:)

[abhigoku10]

@Wilbur529 yup that was my next step , but the video execution uses threading and mutiframe buffer functionality so just trying to check the feasibility on that , but did u face the same issues with the video

[Wilbur529]

@abhigoku10 Sry i did not face this problem. Based on your description, i think the model has no mistake. So just pay more attention to the evalvideo method, and check the postprocess procedure, good luck for u~

[abhigoku10]

@Wilbur529 sure i shall do it , how many iterations you had trained ur mobilenet model , since i trained with 2 lakh iterations the results on the validation set is not that great though the fps is high

[Wilbur529]

@abhigoku10 I let it ran on only one single GPU, so it needed long time for training. At 200 epochs, the mAP are 14.36(box) and 15.02(mask).

[abhigoku10]

@Wilbur529 thanks for the info i shall run it to 200 epochs and share my mAP, anyother architecture u tried besides mobilenet

[Wilbur529]

@abhigoku10 EfficientNets and MobileNet-v3 is coming~~~

[abhigoku10]

@Wilbur529 cool any plans on sharing the reference code

[Wilbur529]

I am reading both of their paper, and i haven't found a suitable pytorch implementation of them~ Once I succeed in my experiment, i will show the modified parts.

[abhigoku10]

@Wilbur529 thanks for the update eagers waiting for this

[pandamax]

I am reading both of their paper, and i haven't found a suitable pytorch implementation of them~ Once I succeed in my experiment, i will show the modified parts.

Looking forward for your update~there is a reference that may be helpful

[abhigoku10]

@Wilbur529 can you share the reference code for the efficient net so that i can test it on different data , what is the fps and accuracy u achieved

[Wilbur529]

@abhigoku10 @dbolya EfficientNet is really efficient!!! I used the implementation and pretrained weights from EfficientNet-PyTorch. To validate the availability, i did two experiments:

1) EfficientNet-B0 has similar Top-5 Acc.(93.2%) with ResNet-50(93.0%) on ImageNet. When i change the backbone of YOLACT to EfficientNet-B0, the convergence rate and mAP between them almost is same. But the total model sizes are YOLACT550-EfficientNet-B0(47MB) and YOLACT550-ResNet-50(120MB).

2) EfficientNet-B4 has similar Top-5 Acc.(96.3%) with SENet(96.2%) on ImageNet, and its parameters size(19M) is less than ResNet-50(26M). When i change the backbone of YOLACT to EfficientNet-B4, i found it may achieve a better performance beyond the previous network. After 55 epochs training, now it have achieved 30.95 mAP on COCO test-dev, and it seems not converge completely. It is important to note that the YOLACT550-EfficientNet-B4 model size is only 101MB.

[abhigoku10]

@Wilbur529 wonderful experimentation and anlaysis , would you be able to share the yolact-efficient code , i would be able to perform certain more experiments , since effcient-b0 model size is an interesting one
THnaks in advance

[abhigoku10]

@Wilbur529 @dbolya i have trained the model with efficient b0 and b4 here is my following observations .

1. Efficient b0 is of model size of 42MB its mAP is similar to Resnet 18 which is of size 72MB but the performance of b0 for multiframe=2 is 20 fps but for resent18 with multiframe=2 is 22 fps for a resolution of 1280x720, which is strange since the model size and params was less but we getting less fps compared to resnet18
2. Similary b4 is of model size of 96MB if of 16fps for multiframe=2 but Resnet 50 which is of size 125MB is of 18fps for multiframe=2
   I am testing on 2080TI gpu system . any suggestions on this behaviour

[Wilbur529]

@abhigoku10 I may suggest you only measure the network inference time:)

[dbolya]

@abhigoku10 @Wilbur529 Yeah if you can, use the included --benchmark mode. Also, that might just be your implementation being slow, but having fewer parameters doesn't necessarily mean you're going to be faster if evaluation is more complex.

[abhigoku10]

@dbolya i shall check up my implementation once again , i agree its not directly proportional that fewer params higher accuracy but is it not higher fps

[zhangbaoj]

What does 'selected_layers': [1, 2, 3] mean?

[dbolya]

@zhangbaoj I replied in your new issue.

[lkj1114889770]

@abhigoku10 can you share your yolact-efficient code?

[Jackerz312]

@Wilbur529 @abhigoku10 can you help me with the efficient backbone? i trained efficientnet-b3 but i only have 30.05 box and 28.13 mask (using pretrained weights). I think my problem is in the config file, can you share that with me pls?

pd: i'm using resnet50 config copy and [3,4,5] selected layers.

Thx :)

[ghost]

@dbolya Any word regarding integrating a new backbone into yolact?

[dbolya]

@deepseek I didn't intend to add any new backbones to the master branch, but I guess if enough people want mobile net or efficient net, I could try. It should be relatively simple to add new backbones yourself if you look at how they're defined in backbone.py and how their configs are set up.

[abhigoku10]

@dbolya i have an efficient net cod but i wanted to review it is it possbile to review it before merging into the branch @Jackerz312 and @deepseek once is get it reviewed i can share it

[dbolya]

@abhigoku10 Can you make a pull request for it? Then we can all review it.

[xscjun]

@Wilbur529
I meet the warning " Warning: Moving average ignored a value of inf " During the training , when I change the backbone of resnet to mobilenetv2 . Did you met this warning ?

[dbolya]

@xscjun I have a patch coming as a result of #222 that might fix that issue for you.

[sdimantsd]

@Wilbur529 ,
Can you please share your EfficientNet backbone?
How can i use it in me network?

[sdimantsd]

What the FPS did you get?
On which GPU?

[sdimantsd]

@IMFinethankyou , can you share you'r config and backbone file?
I don't know how to do it...

[IMFinethankyou]

hi, @sdimantsd, you can check this https://github.com/toandaominh1997/EfficientDet.Pytorch, my backbone file is from this repo. and the config is much like that of the resnet-101 version, except that you need to be careful about the anchors setting.

------------------ 原始邮件 ------------------
发件人: "sdimantsd"<[email protected]>;
发送时间: 2019年12月12日(星期四) 晚上9:23
收件人: "dbolya/yolact"<[email protected]>;
抄送: "Dracula"<[email protected]>;"Mention"<[email protected]>;
主题: Re: [dbolya/yolact] What will happen if I change the backbone network into MobileNet-v2 with FPN? (#27)

@IMFinethankyou , can you share you'r config and backbone file?
I don't know how to do it...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[sdimantsd]

@IMFinethankyou , i tried this one, but YOLACT need more functions/fields like:
add_layer()
layers

[IMFinethankyou]

@IMFinethankyou , i tried this one, but YOLACT need more functions/fields like:
add_layer()
layers

@sdimantsd , add_layer() is used in the backbone construction process, not neccessary in EfficientNet, and the #layers# only used in the load_weights() function for compatability of different version of backbone weights, also not necessary for EfficientNet

[sdimantsd]

OK, i try it.
What do you mean by "anchors setting"? how do i need to change it?
What was the accuracy/map on you'r network?
What was the FPS? on what GPU?
Can you tell me what was the FPS on one of the "default" YOLACT backbone?

Thanx!!

[IMFinethankyou]

OK, i try it.
What do you mean by "anchors setting"? how do i need to change it?
What was the accuracy/map on you'r network?
What was the FPS? on what GPU?
Can you tell me what was the FPS on one of the "default" YOLACT backbone?

Thanx!!

26.8 mAP for B0, 29.6mAP for B3.
GTX 1080Ti, base YOLACT 37 FPS
EfficientNet-B0 41FPS

[sdimantsd]

Thanks!
And what about the anchors? How can i know the best settings?

[abhigoku10]

@IMFinethankyou i performed the same thing but i had got same accuracy for B4 , as @sdimantsd was asking did u re-generate the anchors ?? i used the mobile net anchors . Can you share what changes you have made in the config file

[sdimantsd]

@IMFinethankyou, I don't know what im doing wrong, but i steel can't manage EfficientNet to work as backbone...
I used it as backbone the following config:
efficient_net_config = yolact_base_config.copy({
'name': 'efficient_net',
# 'fpn': None,
# Dataset stuff
'dataset': ws_datasetV1,
'num_classes': len(ws_datasetV1.class_names) + 1,

# Training params
'lr_steps': (280000, 600000, 700000, 750000, 850000, 1000000, 1100000, 1200000, 1300000, 1350000, 1400000, 1450000),
'max_iter': 800000000,


'masks_to_train': 300,
'max_size': 700,

'backbone': efficient_net_backbone.copy({
    'selected_layers': list(range(2, 5)),

    'pred_scales': yolact_base_config.backbone.pred_scales,
    'pred_aspect_ratios': yolact_base_config.backbone.pred_aspect_ratios,
    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': True,  # This is for backward compatability with a bug
}),

})

(i copyed from another backbone, im not shur if it's good)

When i try to train i get the following error:
AttributeError: 'EfficientNetBackbone' object has no attribute 'channels'
in yolact.py line 426:
src_channels = self.backbone.channels

If i set backvone.channels to []
It crash in line 430:
self.fpn = FPN([src_channels[i] for i in self.selected_layers])

Can you please help me?

[IMFinethankyou]

@IMFinethankyou, I don't know what im doing wrong, but i steel can't manage EfficientNet to work as backbone...
I used it as backbone the following config:
efficient_net_config = yolact_base_config.copy({
'name': 'efficient_net',

'fpn': None,

Dataset stuff

'dataset': ws_datasetV1,
'num_classes': len(ws_datasetV1.class_names) + 1,

# Training params
'lr_steps': (280000, 600000, 700000, 750000, 850000, 1000000, 1100000, 1200000, 1300000, 1350000, 1400000, 1450000),
'max_iter': 800000000,


'masks_to_train': 300,
'max_size': 700,

'backbone': efficient_net_backbone.copy({
    'selected_layers': list(range(2, 5)),

    'pred_scales': yolact_base_config.backbone.pred_scales,
    'pred_aspect_ratios': yolact_base_config.backbone.pred_aspect_ratios,
    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': True,  # This is for backward compatability with a bug
}),

})

(i copyed from another backbone, im not shur if it's good)

When i try to train i get the following error:
AttributeError: 'EfficientNetBackbone' object has no attribute 'channels'
in yolact.py line 426:
src_channels = self.backbone.channels

If i set backvone.channels to []
It crash in line 430:
self.fpn = FPN([src_channels[i] for i in self.selected_layers])

Can you please help me?

backbone.channels is a list, contains the channels of the output feature map
for instance: you backbone output a tuple of 5 feature maps:
torch.Size([1, 64, 67, 67])
torch.Size([1, 128, 33, 33])
torch.Size([1, 176, 16, 16])
torch.Size([1, 304, 8, 8])
torch.Size([1, 512, 4, 4])
then the channels should be [64, 128, 176, 304, 512]
backbone.channels is used latter in the FPN module

[IMFinethankyou]

@IMFinethankyou i performed the same thing but i had got same accuracy for B4 , as @sdimantsd was asking did u re-generate the anchors ?? i used the mobile net anchors . Can you share what changes you have made in the config file

I borrowed the config from the base-config, no special changes. the only change i made is set USE_SQUARE_ANCHORS to False.

[sdimantsd]

@IMFinethankyou, thanks, i check it now.
How can i know the backbone output?

[sdimantsd]

also, init_backbone() is not exist :-(

[sdimantsd]

I don't know how you did it, Also backbone_modules is missing...

[sdimantsd]

@IMFinethankyou Can you help me plz?

[IMFinethankyou]

@IMFinethankyou Can you help me plz?

@sdimantsd, you should modify the efficientnet according to the backbone.py, so it can fit into the YOLACT. my connection to github is rediculously slow.

[linshaodan]

@IMFinethankyou , can you share you'r config and backbone file? thks

[IMFinethankyou]

based on the error information， i guess you used the full classification model. the 1000 is the number of imagenet classes.  when you do detection, you should remove the classification head. -------- 原始邮件 --------发件人： linshaodan [e-mail]日期： 2020年1月18日周六 14:08收件人： dbolya/yolact [e-mail]抄送： IMFinethankyou [e-mail], Mention [e-mail]主 题： Re: [dbolya/yolact] What will happen if I change the backbone network into MobileNet-v2 with FPN? (#27)@IMFinethankyou i use efficientnet as backbone ,show this error:
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 256, but got 1-dimensional input of size [1000] instead
can you help me ,please

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[linshaodan]

@abhigoku10 @dbolya EfficientNet is really efficient!!! I used the implementation and pretrained weights from EfficientNet-PyTorch. To validate the availability, i did two experiments:

1. EfficientNet-B0 has similar Top-5 Acc.(93.2%) with ResNet-50(93.0%) on ImageNet. When i change the backbone of YOLACT to EfficientNet-B0, the convergence rate and mAP between them almost is same. But the total model sizes are YOLACT550-EfficientNet-B0(47MB) and YOLACT550-ResNet-50(120MB).
2. EfficientNet-B4 has similar Top-5 Acc.(96.3%) with SENet(96.2%) on ImageNet, and its parameters size(19M) is less than ResNet-50(26M). When i change the backbone of YOLACT to EfficientNet-B4, i found it may achieve a better performance beyond the previous network. After 55 epochs training, now it have achieved 30.95 mAP on COCO test-dev, and it seems not converge completely. It is important to note that the YOLACT550-EfficientNet-B4 model size is only 101MB.

Can you please share your EfficientNet backbone?

[eslambakr]

@dbolya Hello thanks for your amazing work. When I change the backbone to MobileNet by the following the above code which is provided by @Wilbur529, I got this error:

RuntimeError: copy_if failed to synchronize: device-side assert triggered

And sometimes I got Nan and inf in the loss However I am working with the latest version with has this problem was solved by #222 (Note: I got it only when changing the backbone to mobilenet)

@Wilbur529 @abhigoku10 Thank you for providing us by a reference code for MobilnetV2 backbone but unfortunately I doesn't work for me as I mentioned above so did you face the above problems and could help?

Are there any changes than the above provided code?

Thanks in advance.

[eslambakr]

I solved the above issue by lowering the learning rate to 1e-4 instead of 1e-3.
I mentioned the solution to help others who may face the same issue.

But I faced another problem now, the mAP on the validation dataset (I used a custom dataset) as shown in the below figure

Note that: The training is working fine when I used the base backbone, the problem appears only when I changed the backbone to mobilenet.

[eslambakr]

Hello,
@Jackerz312 @Wilbur529 @abhigoku10
could you provide us with your implementation to Efficient-net backbone as It seems I couldn't port it in right way I followed this implementation https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py

I have a couple of questions @dbolya which may help me in porting it in the right manner
1- How should I set this values? will they be affected by changing the backbone?

'selected_layers': [1, 2, 3],
'pred_scales': [[24], [48], [96], [192], [384]],
'pred_aspect_ratios': [[[1.414]]] * 5,

2- Regarding the output I set it like that self.select_idxs = [1, 2, 4, 6]

3- should I take care of any other configuration while porting a new backbone?

For your information currently the loss is stack at certain high value and the mAP is zero

Thanks in advance.

[linshaodan]

@IMFinethankyou @Jackerz312 @Wilbur529 @abhigoku10
I use the efficientnet as the backbone show this errors :
Traceback (most recent call last):
File "train.py", line 532, in
train()
File "train.py", line 322, in train
losses = net(datum)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(input, *kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
raise output
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
output = module(input, *kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(input, *kwargs)
File "train.py", line 147, in forward
preds = self.net(images)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(input, *kwargs)
File "/home/datasets/yolact-master/yolact.py", line 600, in forward
proto_out = self.proto_net(proto_x)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(input, *kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(input, *kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 256 16 3 3, expected input[2, 40, 69, 69] to have 16 channels, but got 40 channels instead

the config as follow:
yolact_EfficientNet_mydataset_config = yolact_base_config.copy({
'name': 'yolact_EfficientNet',

'dataset': my_custom_dataset,
'num_classes': len(my_custom_dataset.class_names) + 1,

'fpn': None,
'max_iter': 120000,
'lr_steps': (60000, 100000),

#'mask_proto_src': 0,

'backbone': EfficientNet_backbone.copy({
    'selected_layers': list(range(1, 4)),
    'pred_scales': [[24], [48], [96], [192], [384]],
    'pred_aspect_ratios': [[[1.414]]] * 5,

    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': False,  # This is for backward compatability with a bug

}),

})
can you help me,please?
Can you please share your EfficientNet backbone code?
thks in advance

[Wilbur529]

I solved the above issue by lowering the learning rate to 1e-4 instead of 1e-3.
I mentioned the solution to help others who may face the same issue.

But I faced another problem now, the mAP on the validation dataset (I used a custom dataset) as shown in the below figure

Note that: The training is working fine when I used the base backbone, the problem appears only when I changed the backbone to mobilenet.

The gap between box mAP and mask mAP is not abnormal. You may find the answer in the author'
s new papar _YOLACT++_.

[Wilbur529]

@IMFinethankyou @Jackerz312 @Wilbur529 @abhigoku10
I use the efficientnet as the backbone show this errors :
Traceback (most recent call last):
File "train.py", line 532, in
train()
File "train.py", line 322, in train
losses = net(datum)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
raise output
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
output = module(input, *kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in *call
result = self.forward(input, kwargs)
File "train.py", line 147, in forward
preds = self.net(images)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in *call
result = self.forward(input, kwargs)
File "/home/datasets/yolact-master/yolact.py", line 600, in forward
proto_out = self.proto_net(proto_x)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in *call
result = self.forward(input, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in *call
result = self.forward(input, *kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 256 16 3 3, expected input[2, 40, 69, 69] to have 16 channels, but got 40 channels instead

the config as follow:
yolact_EfficientNet_mydataset_config = yolact_base_config.copy({
'name': 'yolact_EfficientNet',

'dataset': my_custom_dataset,
'num_classes': len(my_custom_dataset.class_names) + 1,

'fpn': None,
'max_iter': 120000,
'lr_steps': (60000, 100000),

#'mask_proto_src': 0,

'backbone': EfficientNet_backbone.copy({
    'selected_layers': list(range(1, 4)),
    'pred_scales': [[24], [48], [96], [192], [384]],
    'pred_aspect_ratios': [[[1.414]]] * 5,

    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': False,  # This is for backward compatability with a bug

}),

})
can you help me,please?
Can you please share your EfficientNet backbone code?
thks in advance

It seems like you did not change the channels num properly. You can check the first few layers in the protonet.

[Auth0rM0rgan]

Hey @Wilbur529, Have you tried EfficientNet as a backbone? if yes, what performance and FPS have you achieved compare to ResNet50? Are you planning to share your EfficientNet backbone?
Also, I have a question about your MobileNet backbone that what is the intuition behind of selecting these layers select_idxs = [1, 2, 4, 6] not other layers?

Thanks!

[abhigoku10]

@Auth0rM0rgan i was able to implement efficient net as backbone the performance was lower compared to resnet for objects like pedistrains and the fps was higher than resent 50 , i just used the same pred_scales, ascpect ratios similar to mobile net not sure is that the right approach , i could not share the implementation due to firm polices , @Wilbur529 @dbolya can you suggest 'selected_layers': [1, 2, 3],
'pred_scales': [[24], [48], [96], [192], [384]],
'pred_aspect_ratios': [[[1.414]]] * 5,
'use_pixel_scales': False, is this the rigthvalues selected

[linshaodan]

------------------ 原始邮件 ------------------
发件人: "abhigoku10"<[email protected]>;
发送时间: 2020年2月18日(星期二) 晚上11:07
收件人: "dbolya/yolact"<[email protected]>;
抄送: "执着"<[email protected]>;"Comment"<[email protected]>;
主题: Re: [dbolya/yolact] What will happen if I change the backbone network into MobileNet-v2 with FPN? (#27)

@Auth0rM0rgan i was able to implement efficient net as backbone the performance was lower compared to resnet for objects like pedistrains and the fps was higher than resent 50 , i just used the same pred_scales, ascpect ratios similar to mobile net not sure is that the right approach , i could not share the implementation due to firm polices , @Wilbur529 @dbolya can you suggest 'selected_layers': [1, 2, 3],
'pred_scales': [[24], [48], [96], [192], [384]],
'pred_aspect_ratios': [[[1.414]]] * 5,
'use_pixel_scales': False, is this the rigthvalues selected

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[linshaodan]

@abhigoku10
i was able to implement efficient net as backbone，but after training to 1000 steps the loss gradient cannot be reduced.i use efficient-b0 b3 b4, b4 of loss gradient connot be reduced,b0 and b3 of mAP is very low.
please help ！
Thanks in advance！

[abhigoku10]

@linshaodan even i had the same behaviour i had trained for 6lakh iterations , but my mAP was low compared to resnet50 but fps was high

[linshaodan]

@abhigoku10
Yes, I got the same results as you said.
thank you.

[Serge-weihao]

Can you tell me what was the FPS on one of the "default" YOLACT backbone?

OK, i try it.
What do you mean by "anchors setting"? how do i need to change it?
What was the accuracy/map on you'r network?
What was the FPS? on what GPU?
Can you tell me what was the FPS on one of the "default" YOLACT backbone?
Thanx!!

26.8 mAP for B0, 29.6mAP for B3.
GTX 1080Ti, base YOLACT 37 FPS
EfficientNet-B0 41FPS

@IMFinethankyou could you tell me the mAP for val set and test-dev?

[moddent]

@IMFinethankyou Hi,
I modify EfficientDet as EfficientNet backbone,
but I met this:

during training with efficientnet-b0 backbone.
And the loss doesn't reduce after iteration 1000.
The config as shown below:
efficientnet_backbone = resnet101_backbone.copy({
'name': 'EfficientNet',
'path': 'xxx.pth',
'type': EfficientNetBackbone,
'transform': efficientnet_transform,
'selected_layers': list(range(1, 4)),
})

'backbone': efficientnet_backbone.copy({
'selected_layers': list(range(1, 4)),
'use_pixel_scales': True,
'preapply_sqrt': False,
'use_square_anchors': False,
'pred_aspect_ratios': [[[1.414]]] * 5,
'pred_scales': [[24], [48], [96], [192], [384]],
}),

I don't know what goes wrong.
Can you share your config and backbone file or give some suggestions?
thks.