Yet-another-efficientdet-pytorch: train custom dataset

[bitwangdan]

thanks for sharing the code, i trained on my own dataset( six classes), and i changed the coco.yaml, after 10 epochs, the eval result is 0, in training, the loss is normal

[bitwangdan]

@zylo117 is there anything wrong in my settings?

[samson-wang]

Me too. Very weird predictions. Normal loss decrease.

[bitwangdan]

Me too. Very weird predictions. Normal loss decrease.

my prediction box is a negative number in "val2017_bbox_results.json"

[zylo117]

I might release a tutorial about how to train on a custom dataset. stay tune

[bitwangdan]

I might release a tutorial about how to train on a custom dataset. stay tune

thanks

[bitwangdan]

@zylo117 my prediction result is like this, my custom data should be right, i trained in other model works well

[bitwangdan]

hi，This is the result of my training according to the training course you just updated (dataset_shape), what is your training result?

[bitwangdan]

@zylo117

[zylo117]

@zylo117

sorry, it's still being tuned. I will update when it's done

[bitwangdan]

@zylo117

sorry, it's still being tuned. I will update when it's done

good

[zylo117]

it's a bug, so sorry for the trouble. please pull the latest code and try again

[zylo117]

hi，This is the result of my training according to the training course you just updated (dataset_shape), what is your training result?

This is the result after 6 epochs, the bug should be fixed by now. apologize again.
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.726
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.952
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.834
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.757
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.638
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.444
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.776
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.776
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.802
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.702

[samson-wang]

Start working!

[bitwangdan]

@samson-wang hi, Can you train normally? Is it right?

[zizizihua]

After 50 epochs, the trainning loss is less than 0.05, but get bad prediction. What might be the reason？
The picture is written by training process with '--debug=True'.

[zylo117]

After 50 epochs, the trainning loss is less than 0.05, but get bad prediction. What might be the reason？
The picture is written by training process with '--debug=True'.

it is the latest code? the bug hasn't been fixed till 9 hrs ago.

pls share tensorboard loss graph, log, all of your hyperparammeters, etc.

[zizizihua]

it's the latest code. All hyperparammeters are same with default, only changed num_classes.

[zylo117]

it's the latest code. All hyperparammeters are same with default, only changed num_classes.

what is the dataset, coco?
did you train from scratch without pretrained?

you need to provide more info or the minimal code to replicate this issue.

[samson-wang]

@bitwangdan Yes, I managed to train on my own dataset. However, not as good as expected. Still need some tuning. Maybe the hyperparameters not right for my dataset. But the config file support too few params.

@zizizihua It seems that you use a too small learning rate.

[bitwangdan]

it's the latest code. All hyperparammeters are same with default, only changed num_classes.

Is it the result of previous epochs?After a dozen epochs, the effect is better, but it seems to converge a bit slowly

[zylo117]

@bitwangdan Yes, I managed to train on my own dataset. However, not as good as expected. Still need some tuning. Maybe the hyperparameters not right for my dataset. But the config file support too few params.

@zizizihua It seems that you use a too small learning rate.

Did you try running the tutorial about how to train on a custom dataset? It works, so maybe you can find out what actually went wrong by comparing it with yours.

[zizizihua]

Thanks. I ran the tutorial and it did work. I' m running my dataset once again. Expecting good results.

[zheng-2019]

i train custom data content 1 class，but the loss can't turn small
@zylo117 and resume training is not normal,when i resume train,the loss beacma very large

[zylo117]

i train custom data content 1 class，but the loss can't turn small
@zylo117 and resume training is not normal,when i resume train,the loss beacma very large

it's expoding gradients. try 100 times smaller lr

[samson-wang]

I've tried many other one-stage models, none of them worked as good as they claimed. I also tried bifpn and efficientnet in two-stage models, no difference between traditional se-resnet. Maybe more work should be done.

However, according to Kaiming's paper, a resnet50 can get 40+ or 41+ map on coco.

So 模型无好坏，炼丹有高低

[zheng-2019]

------------------ 原始邮件 ------------------
发件人: "WANG Chenxi"<[email protected]>;
发送时间: 2020年4月15日(星期三) 中午11:11
收件人: "zylo117/Yet-Another-EfficientDet-Pytorch"<[email protected]>;
抄送: "郑江红"<[email protected]>;"Comment"<[email protected]>;
主题: Re: [zylo117/Yet-Another-EfficientDet-Pytorch] train custom dataset (#66)

I've tried many other one-stage models, none of them worked as good as they claimed. I also tried bifpn and efficientnet in two-stage models, no difference between traditional se-resnet. Maybe more work should be done.

However, according to Kaiming's paper, a resnet50 can get 40+ or 41+ map on coco.

So 模型无好坏，炼丹有高低

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

ths !!! i try again

[bitwangdan]

@zylo117

这是我50个epoch后的结果，map:42%，the loss is very low，在其他efficientdet的project中，map is 60%, Could it be the problem of anchor size?

[zylo117]

@zylo117

这是我50个epoch后的结果，map:42%，the loss is very low，在其他efficientdet的project中，map is 60%, Could it be the problem of anchor size?

you need to check on loss graph, it might not converge yet.

[bitwangdan]

@zylo117 hi, here




It looks like it's completely converged。
python train.py -c 0 --batch_size 8 --lr 1e-5 --num_epochs 100 --load_weights ./weights/efficientdet-d0.pth
I trained with these parameters

[zylo117]

no. it's just overfitting...

[bitwangdan]

no. it's just overfitting...
hi，Can you give me some advice? My idea is to narrow the learning rate and adjust the rules of learning rate change

[bitwangdan]

@zylo117 hi, How do you see that the model has been overfitting? The convergence seems normal

[cendelian]

hi I also have the same problem.
I just train one class(person), after 70 epoch. I test one img, and prediction is so bad.
my tain command is :
python person_train.py --lr 1e-5 --batch_size 12 --load_weights ./weights/efficientdet-d0.pth

and one img's test result is as below.
there are so many box.
and i think the loss is enough low but not overfitting.

could you help me?

by the way. the person data come from voc2012..

[cendelian]

I add more one test result

I think the anchor ratio is wrong,
but why so many bbox?

anything help is great.

thanks

[zylo117]

1. make sure the anchor config is consistent among training,evaluation,inference

2. make sure image size is the same as training while inferring, which mean force input size must be set to None

3. it's normal that there are false positive boxes, especially in early stage.

4. loss being low doesn't mean no overfitting.

5. person is a difficult class to train while not many folks realize this.

6. add --debug True, then check out test/ folder, there are annotated images during training, it helps visualizing the performance.

7. the right way to judge the model's performance is to run coco_eval

[bitwangdan]

@zylo117
hi, Thanks for your reply，this is the result of my training on my own data with the efficientdet-d1 model, I put the learning rate to 10 times, 1e-4,the map on my dataset is 57% after 15 epochs, close to yolo v3 and other efficientdet projects (63%),I will continue training.this is my loss result, does this look normal?

[zylo117]

@zylo117
hi, Thanks for your reply，this is the result of my training on my own data with the efficientdet-d1 model, I put the learning rate to 10 times, 1e-4,the map on my dataset is 57% after 15 epochs, close to yolo v3 and other efficientdet projects (63%),I will continue training.this is my loss result, does this look normal?

Good job! Congratulations!

Keep training. It looks like still a long way to go.

[zizizihua]

@zylo117 Thanks for your work. Finally it got map 81% by d4 on my dataset. It's a nice performance.

[bitwangdan]

@zylo117 Thanks for your work. Finally it got map 81% by d4 on my dataset. It's a nice performance.

hi, Have you adjusted the lr?

[cendelian]

@zylo117 Thanks for your work. Finally it got map 81% by d4 on my dataset. It's a nice performance.

have you change the anchor ratio? my prediction is so bad

[zizizihua]

@bitwangdan I used default lr 1e-4. The lowest val loss is 0.08.

[bitwangdan]

@bitwangdan I used default lr 1e-4. The lowest val loss is 0.08.
okay,thanks

[zizizihua]

@cendelian no, my dataset is similar to coco, just less class num. Did you set score thresh while inferring image. I found that my eval_json_result contains lots of bbox which score is less than 0.1.

[cendelian]

1. make sure the anchor config is consistent among training,evaluation,inference
   the default anchor ratio come from coco. but i think it's not resonable to get too bad bbox(too long and shin)
2. make sure image size is the same as training while inferring, which mean force input size must be set to None
   -> Yes , they are same
3. it's normal that there are false positive boxes, especially in early stage.
4. loss being low doesn't mean no overfitting.
   -> I have test the 20 epoch. 40 epoch. 70 epoch. they are all bad prediction.
5. person is a difficult class to train while not many folks realize this.
   -> I use the same data to train yolov3, the result is good.
6. add --debug True, then check out test/ folder, there are annotated images during training, it helps visualizing the performance.
   -> I did it. after 3 epoch, the prediction is so bad too.
7. the right way to judge the model's performance is to run coco_eval
   -> I run the coco_eval for my person data. the map is too low....

I will continue to try.

and if you have any suggestion. it great.
thanks again.

[zheng-2019]

------------------ 原始邮件 ------------------
发件人: "zizizihua"<[email protected]>;
发送时间: 2020年4月16日(星期四) 上午10:18
收件人: "zylo117/Yet-Another-EfficientDet-Pytorch"<[email protected]>;
抄送: "郑江红"<[email protected]>;"Comment"<[email protected]>;
主题: Re: [zylo117/Yet-Another-EfficientDet-Pytorch] train custom dataset (#66)

@bitwangdan I used default lr 1e-4. The lowest val loss is 0.08.

—
You are receiving this because you commented.
directly, Reply to this email directly, view it on GitHub, or unsubscribe.

i train my dataset, only contern 1 class,finally,the eval f1 = 0.412

[zheng-2019]

------------------ 原始邮件 ------------------
发件人: "zizizihua"<[email protected]>;
发送时间: 2020年4月16日(星期四) 上午10:22
收件人: "zylo117/Yet-Another-EfficientDet-Pytorch"<[email protected]>;
抄送: "郑江红"<[email protected]>;"Comment"<[email protected]>;
主题: Re: [zylo117/Yet-Another-EfficientDet-Pytorch] train custom dataset (#66)

@cendelian no, my dataset is similar to coco, just less class num. Did you set score thresh while inferring image. I found that my eval_json_result contains lots of bbox which score is less than 0.1.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

i alse train my dataset, only contern 1 class,fanally,the eval f1 = 0.412

[Cli98]

In case of your need to compute anchors, you may have a look here
https://github.com/Cli98/anchor_computation_tool

[mytk2012]

hi I also have the same problem.
I just train one class(person), after 70 epoch. I test one img, and prediction is so bad.
my tain command is :
python person_train.py --lr 1e-5 --batch_size 12 --load_weights ./weights/efficientdet-d0.pth

and one img's test result is as below.
there are so many box.
and i think the loss is enough low but not overfitting.

could you help me?

by the way. the person data come from voc2012..

How can you see your test results?