Yolov3: Interesting Behaviour of Training [NOT a bug/issue]

[orcund]

I just wondered why would that jump happen right after ~200. Epoch.

[glenn-jocher]

@orcund there is an LR scheduler that mulitplies the LR by a gain of 0.1 at 80% and 90% of total epochs:

https://github.com/ultralytics/yolov3/blob/11bcd0f9885ce548c7c123c611921fe63bebe592/train.py#L144

[orcund]

Thanks !!

[gnefihs]

This reminds me of an alternative training schedule. Instead of reducing the steps suddenly at the end, you can achieve the sudden gain in mAP gradually but earlier using the cosine schedule.

It also improves mAP slightly!

https://arxiv.org/abs/1902.04103

[glenn-jocher]

@gnefihs yes this would be a good addition to the repo. We currently use the step scheduler (blue line). If you have time you could submit a PR. The relevant section of the code is here:

https://github.com/ultralytics/yolov3/blob/ddd892dc205c36c029ac9272e92be56fe654415b/train.py#L138-L146

[FranciscoReveriano]

This is something I am looking at right now @gnefihs
Would love to see your ideas.

[gnefihs]

@glenn-jocher i saw that you commented out the burn-in section of your code. did you intentionally leave it out because it doesn't seem to help much?

edit: sorry just saw your comment here. https://github.com/ultralytics/yolov3/issues/238#issuecomment-556840559
How does GIoU remove the need for burn-in?

[glenn-jocher]

@gnefihs the burn-in seemed to help training stability when we used seperate MSE losses for the xywh terms of the boxes, as the wh losses were unbounded, but the switch to GIoU for the box regression loss had a side effect of removing the regression instability, so yes as you said it didn't seem to help much any longer.