Yet-another-efficientdet-pytorch: Anyone has successfully trained on customized dataset ?

[Liu0329]

Many issues seem to have training problems. I just want to know is there anyone has successfully trained on customized dataset. On my own dataset, the training loss jumps between 1000 and 10000. I don't want to freeze head.

Another problem is that COCO's pre-trained weight has 90-class classification layer. Since I found the author use 90 classes ...

[zylo117]

1. there are lots of forks who have trained on custom dataset and got a decent result. It looks like you are suffering overfitting or dataset corruption. If you are new to this, you should check on the tutorial. https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/blob/master/tutorial/train_shape.ipynb

2. yes, coco has 90 classes labels, even though the dataset of 2017 contains only 80 classes.

[Liu0329]

You just repeat claiming it is overfitting given that the training loss is huge or whatever ...
The dataset has no problem since it works well on other detection frameworks.
Can you give me a link of the fork ?

[Liu0329]

The head-only in the tutorial is True. While you cannot require it is forever True.

[betterhalfwzm]

The dataset has no problem since it works well on other detection frameworks.

It is not working on my dataset.The problem is similar to yours.

[zylo117]

Maybe you are training a dataset with only one class, and set class id to 0, but it's wrong. The first class id should be 1, which I mentioned that on readme and the tutorial. And there is no background class

[Zheweiqiu]

I got the similar problem. I rewrote the dataset class and train on my own dataset, but the loss seems to oscillate around 0.2, which I think is wrong since I there is only one class in my dataset. And I dont think it is the overfitting problem because the loss reduced to 0.2 at epoch 2 but remains the same for the next following 20 epochs. Did you figure out what is happening?

[Liu0329]

Maybe you are training a dataset with only one class, and set class id to 0, but it's wrong. The first class id should be 1, which I mentioned that on readme and the tutorial. And there is no background class

I trained on 10 classes, and set the first class id to 1. Both training from scratch and loading pre-trained model do not work.

[zylo117]

@Zheweiqiu , cls loss 0.2 seems ok to me, did you test the mAP?
you can refer to the tutorial, https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/blob/master/tutorial/train_shape.ipynb

This will show you how to train on a custom dataset in 30 minutes.

The final cls loss is around 0.06 at epoch 50, but it's 0.2 at epoch 5, it just takes time to converge.
And the mAP 0.5:0.95 at epoch 50 is 0.7, which is good enough for me.

[Liu0329]

@zylo117 This is training COCO from scratch.

[zylo117]

Is it without any pretrained weights? It will take ages.

But don't panic when cls loss is over 1000 or 10000, if happens when you train from scratch or use a different num_classes. After a few epochs, it will drop.

[Liu0329]

Is it without any pretrained weights? It will take ages.

But don't panic when cls loss is over 1000 or 10000, if happens when you train from scratch or use a different num_classes. After a few epochs, it will drop.

OK. Are the weights you provide trained from scratch ?

[zylo117]

no, it's transferred and finetuned

[Zheweiqiu]

@Zheweiqiu , cls loss 0.2 seems ok to me, did you test the mAP?
you can refer to the tutorial, https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/blob/master/tutorial/train_shape.ipynb

This will show you how to train on a custom dataset in 30 minutes.

The final cls loss is around 0.06 at epoch 50, but it's 0.2 at epoch 5, it just takes time to converge.
And the mAP 0.5:0.95 at epoch 50 is 0.7, which is good enough for me.

Have not tried to test the mAP yet. I achieved 1e-2 level cls loss by following the tutorial on "shape" dataset. For my own dataset, the cls loss remains 0.2 far at epoch 50 so I assume there is something wrong.
You think cls loss 0.2 for a one-class dataset is a normal case? Things will be better if I try to test mAP?

[zylo117]

@Zheweiqiu
Yes, you should test it before making any conclusion. Also, pls share the loss graph if possible.

[zylo117]

@Liu0329 you don't need to modify coco_eval.py.

use this:
python coco_eval.py -p your_project_name -c 2 -w /path/to/your/weights

[ancorasir]

I have successfully trained my custom dataset of 4 classes and 200+ classes with D0.

[firefox1031]

@ancorasir How did you train your custom dataset？Can you share your experience？

[Liu0329]

@zylo117 Now the training loss drops around 1.0. It is still high and the result is still poor on my own dataset. I think the problem is under-fitting instead of over-fitting.
Another interesting problem: When I resume from my own saved model. The loss goes back above 1000. There must be something wrong. Maybe the model loading process ? But there is no warning.
I sincerely recommend you provide a training code that working on datasets beyond COCO. VOC will be a good choice.

[gtgtgt1117]

@Zheweiqiu , cls loss 0.2 seems ok to me, did you test the mAP?
you can refer to the tutorial, https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/blob/master/tutorial/train_shape.ipynb

This will show you how to train on a custom dataset in 30 minutes.

The final cls loss is around 0.06 at epoch 50, but it's 0.2 at epoch 5, it just takes time to converge.
And the mAP 0.5:0.95 at epoch 50 is 0.7, which is good enough for me.

hello @zylo117 ,I have the same problom,but test the Map is 0. what's happen?

[zylo117]

@gtgtgt1117 Can you run the tutorial on your device and get a decent result? If so, just follow the steps. Be sure that your dataset is correctly labeled too.

[gtgtgt1117]

@gtgtgt1117 Can you run the tutorial on your device and get a decent result? If so, just follow the steps. Be sure that your dataset is correctly labeled too.

thank you .i run the tutorial .thatis ok!

[lyn1874]

Same for me. I have successfully trained my own dataset with only one class (person) using D0 (use pretrained coco weight and only fine-tune the head). Training everything from scratch gives me really bad results, but I think the reason is that I don't have enough data and my objects are usually small or tiny, the D0 network which takes a small resolution input than D6 will sometimes just ignore those.