Yet-another-efficientdet-pytorch: training problem

[lockeregg]

Hi, I followed your tutorial to train the coco dataset, after 100 epoch I get this result

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.027
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.012
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.009
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.026
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.017
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.024
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.026
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.021
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.049

the loss decrease during the training process

I changed the batch size = 128 and gpus =8

Did you train with the efficientnet pretrained model?

Thank you!

[zylo117]

pls share your loss graph, training command, log, parameters.

If you have updated to the latest code, then it's likely overfitting

[jinfagang]

Same issue here, what ever epochs train and what dataset I am using, result totally wrong. the result is messed up but loss is very low.

[lockeregg]

[lockeregg]

Same issue here, what ever epochs train and what dataset I am using, result totally wrong. the result is messed up but loss is very low.

hi, what is your pytorch version?

[zylo117]

It's totally overfitting.

[zylo117]

Same issue here, what ever epochs train and what dataset I am using, result totally wrong. the result is messed up but loss is very low.

did you update the latest code?

did you succeed running the tutorial with a decent result?

did you train with --debug True and check on images in test/ folder?

[lockeregg]

It's totally overfitting.
hi,

How do you know it's overfitting?

I have tried all of the output models but I didn't get the same result as yours.

[Li505358678]

I download the code yesterday,My result is the same as the lockeregg,I didn't change any thing and eval with 15 epoch and 25 epoch.But the results both bad

[zylo117]

it really is overfitting.
and my reg loss for d0 is 0.025, val reg loss is 0.035, btw.

so yours weights are far from converging and now overfitting.

[lockeregg]

it really is overfitting.
and my reg loss for d0 is 0.025, val reg loss is 0.035, btw.

so yours weights are far from converging and now overfitting.

Thank you for your reply.

Since many people faced the same problem. I really don't think overfitting is the main problem.

I have started training again, and I'll test every epoch and check the result.

Thank you!

[zylo117]

you'll have to believe me.

because the community has already trained plenty of custom datasets.

and those who claimed that they can't get a good result are the ones who train on coco or voc. These two datasets are very challenging, as we all know. the most common problem is overfitting like yours. So you need to tune the hyperparameters every few hours.

you shouldn't expect this training script can train coco out of the box.

[jinfagang]

Custom dataset trained also fail

[Li505358678]

你得相信我。

因为社区已经培训了大量的自定义数据集。

那些声称自己无法取得好成绩的人就是那些在coco或voc上进行培训的人。正如我们所知，这两个数据集非常具有挑战性。最常见的问题是像你这样的过分适应。所以你需要每隔几个小时调一次超参数。

你不应该指望这个训练脚本可以开箱就训练可可。

I want to know ,how to tune the hyperparameters every few hours ?Just like the Learning rate of the paper?or batch-size is needed to change?

What's more, how many epochs do we need to train on coco?I need fifty minutes to train every epoch. I evaluated on 10,15,25 epoch,they were all wrong

[zylo117]

then why don't you run the tutorial?

https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/blob/master/tutorial/train_shape.ipynb

if tutorial works, it means there might be something worng with your hyperparameters or
it works at first then overfit after a few epochs or dataset corruption.

[Li505358678]

https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/blob/master/tutorial/train_shape.ipynb

I have seen it .I trained on coco2017. I did just like the readme said. I want to get a good result with coco2017. Now the problem is how to tune the hyperparameters?I don't have a good clue. Can you help me out?

[zylo117]

try lower lr and --optim sgd

[zylo117]

here post my result of the tutorial. If you get the same result as I did. Then you can compare the code with yours and find out the root causes.

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.765
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.913
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.837
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.814
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.645
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.467
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.820
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.823
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.861
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.731

[Li505358678]

--optim sgd

thanks for your help ,I will try it. and the result you given is for which?d0?or what?Why it so high? = -、I thought 0.33 would be good。and when can I use sgd?I'm just getting started. I don't know a lot about it

[zylo117]

it's the result of tutorial about how to train on a custom dataset.

yes, it's d0

[Li505358678]

it's the result of tutorial about how to train on a custom dataset.

yes, it's d0

I may be wrong. What I want to say is I trained on coco2017. My results may not be close to yours.The best result is only 0.33

All to all,thank you for your help! I will try what you said

[Li505358678]

I failed，@lockeregg .Do you have good result?if you do ,how do you do?I tune the hyperparameters for every 5 epoch. the result is also bad

[mnslarcher]

you'll have to believe me.

because the community has already trained plenty of custom datasets.

and those who claimed that they can't get a good result are the ones who train on coco or voc. These two datasets are very challenging, as we all know. the most common problem is overfitting like yours. So you need to tune the hyperparameters every few hours.

you shouldn't expect this training script can train coco out of the box.

Hi @zylo117, can you explain better this part? in the paper the authors doesn't mention this manual training process for COCO. From what I read, they use SGD with momentum 0.9 and weight decay 4e-5, they linearly increase the learning rate in the first epoch and then they use cosine annealing. What I'm missing? I need to assume something that is not present in the paper but it's a standard practice in this field?

[zylo117]

Hi, @mnslarcher. The training part is not ported from the official repo, nor the paper. Because they are using TF and lots of TPUs, which are totally different from pytorch and GPUs.

From my experience, hyperparameters, especially learning rate even if they are using the same optimizer, shouldn't be copied from a different framework, otherwise it will usually cause gradidents exploding.

The tuning can be done automatically, of course. But it's not a part of this training script.

There are quite a few folks who have successfully trained on their custom datasets, so the training script works fine, I think. If they can't get a good result, it's likely they are still underfitting or already overfitting, or their datasets could be corrupted. These three are the most common issues I've seen.

[mnslarcher]

Hi, @mnslarcher. The training part is not ported from the official repo, nor the paper. Because they are using TF and lots of TPUs, which are totally different from pytorch and GPUs.

From my experience, hyperparameters, especially learning rate even if they are using the same optimizer, shouldn't be copied from a different framework, otherwise it will usually cause gradidents exploding.

The tuning can be done automatically, of course. But it's not a part of this training script.

Thanks for the answer @zylo117! so the main problem is that changing framework the hyperparameters may no longer be optimal, some tuning is needed (but that would be needed in any case if one change the dataset so is not a big deal). Given that they use a pretty standard training procedure similar to many other papers, I image that one can obtain the same with this implementation after finding the correct value for the learning rate, weight decay etc. without changing completely the procedure itself, I'm correct?

[zylo117]

No, it's not just 'not optimal', it may be totally not acceptable in some cases.

For example, you can refer to matterport/Mask_RCNN, as they mentioned in readme.

Learning Rate: The paper uses a learning rate of 0.02, but we found that to be too high, and often causes the weights to explode, especially when using a small batch size. It might be related to differences between how Caffe and TensorFlow compute gradients (sum vs mean across batches and GPUs). Or, maybe the official model uses gradient clipping to avoid this issue. We do use gradient clipping, but don't set it too aggressively. We found that smaller learning rates converge faster anyway so we go with that.

So the final learning rate will be totally different if you switch to another framework.

[mnslarcher]

Sounds clear, thanks for the example!

[Li505358678]

@zylo117 I see the paper said they employed ImageNet-pretrained EfficientNets as the backbone network when they train on coco2017. Do we need the same net to load it? I can't load the ImageNet-pretrained EfficientNets which I download from the paper

[zylo117]

@zylo117 I see the paper said they employed ImageNet-pretrained EfficientNets as the backbone network when they train on coco2017. Do we need the same net to load it? I can't load the ImageNet-pretrained EfficientNets which I download from the paper

The backbone's key is been changed because of the samepadding conv. If you really need the backbone, you can save the state_dict of the model.backbone_net as backbone weights

[lockeregg]

I failed，@lockeregg .Do you have good result?if you do ,how do you do?I tune the hyperparameters for every 5 epoch. the result is also bad

No. I am now trying to load efficientNet pretrained weight and restart training again.

[Li505358678]

I failed，@lockeregg .Do you have good result?if you do ,how do you do?I tune the hyperparameters for every 5 epoch. the result is also bad

No. I am now trying to load efficientNet pretrained weight and restart training again.

I want to do it ,too. But how it load the efficientNet pretrained weight？I download the efficientNet pretrained model of tensorflow. this code can't load it? Do you have efficientNet pretrained weight of pytorch? Give a link please.Thank you very much!!

[factplay1]

I'm also trying to train EfficientDet-d0 on COCO.
Originally started another issue here: https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/issues/122

But I'll follow up on this issue as it seems some other fellows have the same problem.

Summary of the observations so far:

1. Trained using train.py using the default set of parameters:
   Overfitted. Had jobs on 1, 2, 4, 8 GPUs, with batch sizes from 12, 24, 32, 48, 96, 128, 256.
   Tuned learning rate from 4e-5, 1e-4, 4e-4, 8e-4, 1e-3, 4e-3
   Incorporated gradient clipping, and weight decay.
   This was with Adamw so far.

The best validation total loss so far I achieved is 0.404.
Zylo says this should be down to 0.25 or so for a good training job.

1. Also tried to continue training from the best job after the 0.404 loss, using SGD, gradient clipping, weight decay 4e-5, lr=8e-4. After like 10+ epochs improved to 0.402, but LR dropped to 8e-7, so i don't think it's gonna get any better.

2. Now training with SGD from the beginning. Will write here on how it goes.

I just observed that the training code here is not using any pretrained imagenet weights for the EfficientNet backbone. Is that correct?

[jinfagang]

@factplay1 I believe not traning with a pretrained backbone is the key. It maybe worked for small dataset but not work for large scale data.

[factplay1]

@jinfagang
I agree. That's very likely the case.
Have you tried using pretrained imagenet weights?

[Cli98]

I'm also trying to train EfficientDet-d0 on COCO.
Originally started another issue here: #122

But I'll follow up on this issue as it seems some other fellows have the same problem.

Summary of the observations so far:

1. Trained using train.py using the default set of parameters:
   Overfitted. Had jobs on 1, 2, 4, 8 GPUs, with batch sizes from 12, 24, 32, 48, 96, 128, 256.
   Tuned learning rate from 4e-5, 1e-4, 4e-4, 8e-4, 1e-3, 4e-3
   Incorporated gradient clipping, and weight decay.
   This was with Adamw so far.

The best validation total loss so far I achieved is 0.404.
Zylo says this should be down to 0.25 or so for a good training job.

1. Also tried to continue training from the best job after the 0.404 loss, using SGD, gradient clipping, weight decay 4e-5, lr=8e-4. After like 10+ epochs improved to 0.402, but LR dropped to 8e-7, so i don't think it's gonna get any better.
2. Now training with SGD from the beginning. Will write here on how it goes.

I just observed that the training code here is not using any pretrained imagenet weights for the EfficientNet backbone. Is that correct?

@factplay1 So are you suggesting that tuning learning rate will not help?

[Cli98]

@factplay1 I believe not traning with a pretrained backbone is the key. It maybe worked for small dataset but not work for large scale data.

@jinfagang It depends. If your categories heavily overlap with coco, why not? BTW coco itself is not small

[Cli98]

it really is overfitting.
and my reg loss for d0 is 0.025, val reg loss is 0.035, btw.
so yours weights are far from converging and now overfitting.

Thank you for your reply.

Since many people faced the same problem. I really don't think overfitting is the main problem.

I have started training again, and I'll test every epoch and check the result.

Thank you!

@lockeregg I agree with you. It's not overfitting, clearly.

[Li505358678]

Does someone have a good result?If you do,can you share the training process please?

[factplay1]

Best I have got so far is val loss of 0.33.
With using pre-trained imagenet weights & latest code.
Still not good enough.
Still working on it, will post here if I get good results.

[Li505358678]

Best I have got so far is val loss of 0.33.
With using pre-trained imagenet weights & latest code.
Still not good enough.
Still working on it, will post here if I get good results.
How do you use the pre-trained imagenet weights? load with .pth?Where did you get it?I have weights of tensorflow.How to use it to train?or I need to load another?

[xiaoshuyun]

Best I have got so far is val loss of 0.33.
With using pre-trained imagenet weights & latest code.
Still not good enough.
Still working on it, will post here if I get good results.
How do you use the pre-trained imagenet weights? load with .pth?Where did you get it?I have weights of tensorflow.How to use it to train?or I need to load another?

hello~
now,do you get good results?I still get bad results in my datasets.Can you teach me?

[factplay1]

So I concluded that the code, "as is", does not reproduce COCO training from scratch (efficientdet-0).

Here is what I tried:

• Around 220+ training jobs with different configurations
• hyper-parameter tuning
• 1,2,4,8 GPU cases with various ranges of batch size
• Tried making some changes in the code so it can load pretrained efficientnet-d0 weights. This improved the loss, but still not low enough - lowest it got is 0.305, and even that gives low mAP (4%)
• Tried different kinds of optimizers, different kinds of LR scheduling techniques
• Tried 2-step training, where you train with adamw first, then try to further finetune with sgd
• Tried with the latest code as well as throughout the past 3 weeks or so
• Tried random seed vs fixed seed
• Tried different kinds of pretrained weights

I concluded that for custom datasets it works, but not for COCO training from scratch for research purposes.

[Cli98]

So I concluded that the code, "as is", does not reproduce COCO training from scratch (efficientdet-0).

Here is what I tried:

• Around 220+ training jobs with different configurations
• hyper-parameter tuning
• 1,2,4,8 GPU cases with various ranges of batch size
• Tried making some changes in the code so it can load pretrained efficientnet-d0 weights. This improved the loss, but still not low enough - lowest it got is 0.305, and even that gives low mAP (4%)
• Tried different kinds of optimizers, different kinds of LR scheduling techniques
• Tried 2-step training, where you train with adamw first, then try to further finetune with sgd
• Tried with the latest code as well as throughout the past 3 weeks or so
• Tried random seed vs fixed seed
• Tried different kinds of pretrained weights

I concluded that for custom datasets it works, but not for COCO training from scratch for research purposes.

Thank you @factplay1. As far as I can tell, it may work for customer dataset, but not as good as expected.

[Cli98]

In case of your need to compute anchors, you can have a look here
https://github.com/Cli98/anchor_computation_tool

[Wanjpeng]

@factplay1 ,So did you finally get a similar result as this repo's Introduction said with EfficientNet pretrained weights on COCO2017?