Yet-another-efficientdet-pytorch: advice on training a custom dataset

[kaishijeng]

If my dataset has different # of classes than coco dataset, can I still use:

train efficientdet-d2 on a custom dataset with pretrained weights

with batchsize 8 and learning rate 1e-5 for 10 epoches

Or I need to use this:

with a coco-pretrained, you can even freeze the backbone and train heads only

to speed up training and help convergence.

[zylo117]

you don't need to worry about that. It's been taken care of. you can load pretained weights even though you have different number of classes.

https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/blob/master/train.py#L115

[zylo117]

I think I misunderstood your question, I recommend the latter one.

Train head first, see if the performance is ok. If not, unfreeze the backbone and resume training.

[kaishijeng]

I am not able to get any meaning result with finetune even with coco dataset after 10 epoches with or without keep_head which is a surprise.
I would imagine the result should be quite decent when I use your pretrained d2 weights and finetune with coco dataset. The reason I am experimenting this for sanity check because I can't get any meaningful result with my own datatset.

Any guidance? Can you check on your side whether you can get reasonable detection results after 10 epoches even if you finetune with coco dataset?

[zylo117]

Did you mean that you finetune D2 with coco but got bad result? What is your hyperparameters, such as learning rate?
And how is your training loss/ val loss?
If you're using lr greater than 1e-8, it's very likely that you are suffering overfitting. So the result will be bad.

[kaishijeng]

I am using lr=1e-5.
I have started a finetuning experiment and it is at 4th epoch now. Val loss for 3rd epoch is:
Val. Epoch: 3/40. Classification loss: 0.21641. Regression loss: 0.71791. Total loss: 0.93433
My observation is Regression loss seems to stay at 0.6-0.7 range for many epochs. Not sure why

[zylo117]

I am using lr=1e-5.
I have started a finetuning experiment and it is at 4th epoch now. Val loss for 3rd epoch is:
Val. Epoch: 3/40. Classification loss: 0.21641. Regression loss: 0.71791. Total loss: 0.93433
My observation is Regression loss seems to stay at 0.6-0.7 range for many epochs. Not sure why

And the training loss? How is that? Can you share your D2 evaluation result?

[kaishijeng]

I will wait for 10 epoch completed and do D2 eval result.
Below is trainiing loss and Val loss of the 4th Epoch:
Step: 49284. Epoch: 4/40. Iteration: 9857/9857. Cls loss: 0.08769. Reg loss: 0.54299. Total loss: 0.63068: 100%|█| 9857/985
Epoch 4, lr 1e-05
Val. Epoch: 4/40. Classification loss: 0.21514. Regression loss: 0.70011. Total loss: 0.91525

Below is detection result using 4th epoch trained weight and set threshold at 0.4 instead of 0.2:

[zylo117]

That's just overfitting, the training loss is much lower than you val loss. You lr is way too high.

[kaishijeng]

the training loss is much lower than you val los
Why do you think it is overfitting? The delta of these 2 losses is 0.28:
0.63 vs 0.91

I will change lr to 1e-08 after 10 epoches completed

[kaishijeng]

The thing bothers me is cls loss seems to continue decrease, but Reg loss is staying at 0.5-0..9 range and it doesn't decrease.

[zylo117]

the training loss is much lower than you val los
Why do you think it is overfitting? The delta of these 2 losses is 0.28:
0.63 vs 0.91

I will change lr to 1e-08 after 10 epoches completed

just restart finetuning, once it's overfit, it's hard to un-onverfit it.
the weights is already useless by this point.
and it's not just 0.28, it's 0.28 / 0.91 = 31%, the calculation is not accurate but it means the overfitting is severe

[ancorasir]

The results above doesn't look like overfitting. It's way too wrong ...

I met a similar problem training d0 on my custom dataset. The loss is decreasing well and looks alright but the prediction using efficientdet_test.py doesn't make any sense (not even close). I'm using one of the training images to do the test.

Any suggestions? Have you tried to use the code to train a different dataset other than coco? Thank you@zylo117

[ancorasir]

@kaishijeng , have you tried to inference an image from the training dataset to see if the model give reasonable predictions?

[zylo117]

The results above doesn't look like overfitting. It's way too wrong ...

I met a similar problem training d0 on my custom dataset. The loss is decreasing well and looks alright but the prediction using efficientdet_test.py doesn't make any sense (not even close). I'm using one of the training images to do the test.

Any suggestions? Have you tried to use the code to train a different dataset other than coco? Thank you@zylo117

1. Did you set force_input_size to None?
2. How do you know it's not overfitting? Any proof? Tensorboard/log that contains training/val loss?
3. I'm training a custom dataset myself, the result seems fine.
4. It means nothing by inferring one image, you need to run coco_eval to get a fair evaluation.

[ancorasir]

@zylo117 , Thanks for the quick reply. I have set force_input_size to None. My custom dataset is very simple and there is only one bounding box per image.

The training loss is shown below and it seems at least the model should have learned the training data well.
CUDA_VISIBLE_DEVICES=2 python train.py -c 0 --batch_size 8 --num_workers 8 --lr 1e-5 \
--num_epochs 40 \
--project custom \
--save_interval 4000 \
--load_weights weights/efficientdet-d0.pth \

Step: 70598. Epoch: 39/40. Iteration: 1764/1765. Cls loss: 0.00240. Reg loss: 0.00279. Total loss: 0.Step: 70598. Epoch: 39/40. Iteration: 1764/1765. Cls loss: 0.00240. Reg loss: 0.00279. Total loss: 0.Step: 70599. Epoch: 39/40. Iteration: 1765/1765. Cls loss: 0.00178. Reg loss: 0.00168. Total loss: 0.Step: 70599. Epoch: 39/40. Iteration: 1765/1765. Cls loss: 0.00178. Reg loss: 0.00168. Total loss: 0.

I inferred 20 images from the training set. The classes are all correct with probability 1 but the bounding box are totally wrong. I set iou_threshold = 0.01 so that nms only gives one bounding box.

I have used detectron2 to train the same dataset and it works well. Detectron2 also used the same aspect ratios [0.5, 1.0, 2.0]. So I seems the problem is not caused by the default anchors ratio.

Any suggestions? By the way I use kmeans to find 9 anchors ratios as following:
array([[1.16266418],
[5.13670823],
[2.69625205],
[7.05724746],
[1.5651086 ],
[0.70979814],
[3.58995448],
[9.6087586 ],
[2.05010729]])

[kaishijeng]

@ancorasir

have you tried to inference an image from the training dataset to see if the model give reasonable predictions?
Yes, I got the same issue. Inference generates useless results if I use my own trained weight.
I have tried 3 approaches and none of them works:
1) Finetune with head_only with my dataset
2)Finetune without head_only with my dataset
3)Train D0 with COCO from scratch

The only time I am able to get reasonable result is when I finetune D0/D2 with COCO dataset with very small values of learning rate, like 1e-7.
So I am just wondering whether @zylo117 has finetuned with another dataset successfully or training COCO from the scratch with good result. Would like to reproduce what he has done for sanity check with my environment

[zylo117]

It depends on your network, if you are using two-stage detection network like RCNN, anchors doesn't matter like one-stage does. Your anchors ratio is obiviously different from coco's, you should change the anchor config to fit you dataset.

If anchor config is wrong, the loss will be low, because lots of gt boxes are filltered out.
Maybe tring this one

anchors_ratios: '[(3.0, 1.0), (3, 0.7), (9, 1.4)]'

[kaishijeng]

@zylo117

I am trying to train to reproduce D0 with COCO dataset from a scrach, but also got total wrong bbox, ie BBox is detected without any relationship to an object in an image.
This should not have anything to do with anchor_ratios.
Have you trained D0 with COCO dataset from a scrach and got good results. If yes, plese share your hyper parameters of your training

Thanks

[zylo117]

Please share your

@zylo117

I am trying to train to reproduce D0 with COCO dataset from a scrach, but also got total wrong bbox, ie BBox is detected without any relationship to an object in an image.
This should not have anything to do with anchor_ratios.
Have you trained D0 with COCO dataset from a scrach and got good results. If yes, plese share your hyper parameters of your training

Thanks

Please share loss graph, including training loss, val loss, logs.

[ancorasir]

I found in efficientdet_test.py line 53:
model = EfficientDetBackbone(compound_coef=compound_coef, num_classes=len(obj_list))

The inference model does not use the anchor ratio in the configuration yml file. I add parameters here:
model = EfficientDetBackbone(compound_coef=compound_coef, num_classes=len(obj_list),
ratios=[(3.0, 1.0), (3, 0.7), (9, 1.4)], scales=[2 * 0, 2 * (1.0 / 3.0), 2 ** (2.0 / 3.0)])

I tried to use the suggested anchor ratio and the problem is still there. The bbox prediction is even worse than use the d0 checkpoint you provided directly without any training and use the default coco anchor ratio. The d0 checkpoint you provided is able to detect the location most of the time although the classification may not right. Does this mean it is ok to use the default anchor ratio?

I guess I should change the anchor scale as well. Since there is only one object in the image, so the bounding box could occupy quite a large portion of the image. For example, the width of the bbox is 70% of the image width. Any suggestions on the anchor scales? Thanks very much!
@zylo117

[zylo117]

you need train the model with the new anchor ratios, not just inferring.

[lihuang3]

The results above doesn't look like overfitting. It's way too wrong ...

I met a similar problem training d0 on my custom dataset. The loss is decreasing well and looks alright but the prediction using efficientdet_test.py doesn't make any sense (not even close). I'm using one of the training images to do the test.

Any suggestions? Have you tried to use the code to train a different dataset other than coco? Thank you@zylo117

@ancorasir @kaishijeng Any update? I got the same issues with fine-tuning. I tested the trained weights on training set images. They were just wrong, not overfitting.

[kaishijeng]

@lihuang3

No, all of my trainings (finetuing or training from the scratch, even with COCO dataset) are failed, ie, bbox results are totally wrong. At this point, I will wait for any one who has succeeded in training before I spent more time on it. I believe there is something wrong with training code because I found several people have the same problem.
It will be nice @zylo117 can release his training script and environmentto us so we can reproduce his training results to avoid this trial and error. By the way, I don't have training issue with detectron2 by following its instruction with the same of my own dataset.
Thanks,

[ancorasir]

you need train the model with the new anchor ratios, not just inferring.

I did use the new anchor ratio for both training and inferring
@zylo117

[ancorasir]

So far no good news. I have tried training with or without fixed backbone, the classification is always correct but the bbox regression is far from reasonable although the regression loss is decreasing as normal.
@lihuang3

[zylo117]

I might release a tutorial about how to train on a custom dataset.

[ancorasir]

I might release a tutorial about how to train on a custom dataset.

@zylo117 That will be great. Thanks for the contribution.

[zylo117]

sorry for the training troubles, there's a bug in loss function. please pull the latest code and give it a try.

[ancorasir]

@zylo117 , Thanks for the update. I have trained my custom dataset. Now the model is able to learn the classification and the height of the bbox. But failed to learn the width of the bbox. In other words the predicted bbox are all correct in height but not very tight in width. Is it possible there might be some bug in the loss regarding to the width of the bbox?

[zylo117]

i don't think so. can you infer more images and run a coco style evaluation?

[ancorasir]

@zylo117 Thanks very much. I forgot to change back to the default anchor ratio while inferring. Now everything is good. Thanks for the excellent work! Great job!

[kaishijeng]

@zylo117

With your recent loss fix, I am able to get reasonable results with tuning of D2 network for my own datasets. However, I notice small objects are missing sometimes. Will a bigger network, like D5 improve small object detection? Any suggestion?

Thanks,

[zylo117]

@zylo117

With your recent loss fix, I am able to get reasonable results with tuning of D2 network for my own datasets. However, I notice small objects are missing sometimes. Will a bigger network, like D5 improve small object detection? Any suggestion?

Thanks,

Theoretically yes, but the cost is also expensive

[claire-s11]

@zylo117

I am trying to train to reproduce D0 with COCO dataset from a scrach, but also got total wrong bbox, ie BBox is detected without any relationship to an object in an image.
This should not have anything to do with anchor_ratios.
Have you trained D0 with COCO dataset from a scrach and got good results. If yes, plese share your hyper parameters of your training

Thanks
Hi @kaishijeng , I'm also trying to train D0 on COCO2017 with pretrained EfficientNet 1 model, but my mAP is so bad after the 12th epoch. Could you share your training log if you trained it as well? Thank you!

[zylo117]

d0 should be trained with B0 backbone, not B1

[claire-s11]

@zylo117 Sorry, b0, typo