Keras-yolo3: the loss is around 30, and always 0 box was found when inference

[perry0418]

1 、the first time , I start to the training my own dataset without loading the pretrained weight, because the classes were totally different with COCO. After 500 epoch training, I got a model with val loss of 30, however the model perform pretty good when testing.
2、 This time, I use the same way without loading the pretrained weight to training my own dataset about 700 images. when training comes to 80 epoch, the val loss arrive at 24 and seems does not continue to decrease. So I test the saved weight, the result was not good. There are always 0 box found when inference.

By the way I also tried to training this dataset with loading the pretrained weight, but the val loss was also around 30 and 0 box found when inference. However, my colleague chooses the C language code version YOLO V3 to train the same dataset, and getting a not bad result.

who have the similar experience and how can I reduce the val loss ？

[oak-tree]

@perry0418

1. I would start with verify your data is being loaded properly and that you are normalized the data with inference at the same way as when training.
2. It might that you colleague started also from pretrained network?

[perry0418]

@oak-tree
the problem has been solved by changing the learning rate schedule and continue to train the dataset. The ReduceLROnPlateau was set to reduce the learning rate with patience 10 epoch, however that results the learning rate decreases too fast! In the end the learning rate become 10^-19，that is why the loss cannot reduce anymore.
I start to fix the learning rate as 0.0001 to train with 300 epoch and choose the best checkpoint manually, then change the leaning rate to 0.00001 and load the model to continue training. Finally, the loss reduce to 13 with 700 epoch training.

[enoceanwei]

@perry0418

Sorry to disturb you.

I am suffering a same problem, when I train on my own dataset.

I use the pretrain weight which comes from the weights transferred command. I finished the stage-1 training(50 epoch), however, the new model can not detect anything (0 box for a image).

Could you give me some suggestion about it?

By the way, If I train on my own dataset with the pretrain weights, do I need to changed the 'yolo3.cfg'? such as I need to changed the class from 80 to my real classes numbers?

I am looking forward to your reply, many thanks.

Kind regards

Wei

[perry0418]

@enoceanwei
Obviously, 50 epoch is not enough. when the loss reduce to around 10, your model may have a not bad performance. you can fix the learning rate and training longer ,just be patient.
The yolo3.cfg is used to extract the pretrained weights from darknet. Once you got the pretrained weight, there is no need to do with .cfg files. If you want to training your own dataset, all you need to do is to modify codes in the train.py. by the way, now I turn to Pytorch, I believe Pytorch is more suitable for you.

[enoceanwei]

@perry0418

First of all, thank you for your reply, many thanks.

I will try it, which is follow your suggestion.

I have a new question about the pretrain weights, Didn't you use pretrain during the training? in other words, if I want to train a model on my own dataset, I only do it with a command line(python convert.py -w yolov3.cfg yolov3.weights model_data/yolo_weights.h5) to generate the pretrain model for keras version, and then I can directly use the .h5 file without any revised in the 'yolov3.cfg' file?

Kind regards

Wei

[universewill]

@enoceanwei
Obviously, 50 epoch is not enough. when the loss reduce to around 10, your model may have a not bad performance. you can fix the learning rate and training longer ,just be patient.
The yolo3.cfg is used to extract the pretrained weights from darknet. Once you got the pretrained weight, there is no need to do with .cfg files. If you want to training your own dataset, all you need to do is to modify codes in the train.py. by the way, now I turn to Pytorch, I believe Pytorch is more suitable for you.

My loss is 10, but still get very bad predict result. Even on the train data.
I do not know why

[YHL04]

@universewill I’m facing the same problem, val loss around 20, have you found a solution?