Darknet: Incremental update to dataset

[ssrinivasan]

Alexy ,

Thanks for this wonderful effort . I am curious whether YOLO v3 has an option to incrementally train for new data-sets .

Say for example I have a base dataset (which is huge and training takes over 14 hours on a single GPU machine)which is performing with a MAP over 90 percent . I want to give new training samples (Kind of Reinforcement/Online Learning) . This incremental dataset is really small when compared to original dataset . Can I utilize the existing weight file and train the relatively small incremental dataset without retraining on base dataset plus incremental dataset . This will give me a ability to iterate faster .

If at all this is possible is there are any known issues ? Or do you suggest a full retrain ?

Note : There will be no difference in number of classes . The activity involves only providing additional data .

Thanks for you help

[AlexeyAB]

Yes, you can:

• if you train for the same classes - you can train by using OLD_dataset+NEW_dataset, by using old trained yolov3_obj_last.weights and by using ./darket detector train obj.data yolov3_obj.cfg yolov3_obj_last.weights

• if you train for other classes - you can train by using NEW_dataset, by using old pre-trained yolov3_obj_last.weights that is partially trimmed to yolov3_old_obj.conv.81 file:
  ./darknet partial cfg/yolov3_old_obj.cfg yolov3_old_obj.weights yolov3_old_obj.conv.81 81 - 81 is number of un-changed layers
  and by using ./darket detector train obj.data yolov3_new_obj.cfg yolov3_old_obj.conv.81

More about partial: https://github.com/AlexeyAB/darknet/blob/master/build/darknet/x64/partial.cmd

[ssrinivasan]

Thanks a ton for the quick response . Really appreciate it .

[silviasanna]

Hi, I have a problem with the backup file. I've created the folder backup, put the correctly path in the obj.data file and just in the first iteration it made me the .weights file. For some errors I had to retrained so I've deleted the .weights file, made another backup directory but I haven't the file. What is the problem? Thanks

[ManzarIMalik]

@silviasanna, hello.

Please post in the relevant issue, however, if you have the last.weights file you can resume training your data.

[ManzarIMalik]

Yes, you can:

• if you train for the same classes - you can train by using OLD_dataset+NEW_dataset, by using old trained yolov3_obj_last.weights and by using ./darket detector train obj.data yolov3_obj.cfg yolov3_obj_last.weights
• if you train for other classes - you can train by using NEW_dataset, by using old pre-trained yolov3_obj_last.weights that is partially trimmed to yolov3_old_obj.conv.81 file:
  ./darknet partial cfg/yolov3_old_obj.cfg yolov3_old_obj.weights yolov3_old_obj.conv.81 81 - 81 is number of un-changed layers
  and by using ./darket detector train obj.data yolov3_new_obj.cfg yolov3_old_obj.conv.81

More about partial: https://github.com/AlexeyAB/darknet/blob/master/build/darknet/x64/partial.cmd

@AlexeyAB, If we have to add new classes should we only have paths to new classes dataset in train.txt and test.txt or will it be paths of old classes dataset + new classes dataset? Please confirm.

[AlexeyAB]

@ManzarIMalik If you want to detect old+new classes, then you should continue training with old+new images/pathes/classes/...

[ManzarIMalik]

@ManzarIMalik If you want to detect old+new classes, then you should continue training with old+new images/pathes/classes/...

So If I want to add new classes along with old classes, I need to do:

1. Partial Trim the weight file using: ./darknet partial cfg/yolov3_old_obj.cfg yolov3_old_obj.weights yolov3_old_obj.conv.81 81More then half million and more then 2 millions people visited Tharparkar in last 10 days and 2 months respectively, After recent rains in desert it’s perfect time to visit Thar.
2. Change Filters, Classes, Anchors in new cfg file.
3. Do I need to change max_batches and steps in new cfg file, for instance I want to add 1 new class on 18000(final.weights) so what max_batch should I write?
4. Train and Test files paths of old classes datatset + new class dataset.

Am I going in right direction? @AlexeyAB

[ManzarIMalik]

@AlexeyAB,

I have followed this procedure to train pre-trained weights with one additional class.

• if you train for other classes - you can train by using NEW_dataset, by using old pre-trained yolov3_obj_last.weights that is partially trimmed to yolov3_old_obj.conv.81 file:

But there is nothing happening, I don't see any weights coming, not even convolution net of YOLO in shell. What could be the issue?

I added path of old classes data + new class data as you said.

@ManzarIMalik If you want to detect old+new classes, then you should continue training with old+new images/pathes/classes/...