Darknet: reinforcement learning on object detection

[anguoyang]

Hi@AlexeyAB, I have a idea maybe a little bit crazy:
is it possible to add reinforcement learning on yolo v3? i.e, retrain all the error-inference images and add something like reward/punishment mechanism to improve accuracy?

[AlexeyAB]

@anguoyang Hi, try to check, does it increase mAP on some datasets.

1. Update your code from GitHub

2. un-comment these lines: https://github.com/AlexeyAB/darknet/blob/032acd408acd119a5bd5c132765fcdd6caedea6b/src/detector.c#L727-L734

3. and do:
   ./darknet detector map obj.data cfg/yolov3-obj.cfg backup/yolov3-obj_10000.weights -thresh 0.25

You will get file reinforcement.txt that contains list of images with errors (FP or FN) during detection for the specified -thresh 0.25.

1. then just set train=reinforcement.txt in the obj.data and continue training as usual:
   ./darknet detector train obj.data cfg/yolov3-obj.cfg backup/yolov3-obj_10000.weights

---

By default it stores images with more than 0 errors, but you can change it here: https://github.com/AlexeyAB/darknet/blob/032acd408acd119a5bd5c132765fcdd6caedea6b/src/detector.c#L734

[anguoyang]

@AlexeyAB thank you so much

[anguoyang]

I will try to test on this reinforcement feature and come back to you

[AlexeyAB]

@anguoyang Hi, did you get any good results?

[anguoyang]

@AlexeyAB, I will test the result on reinforce ment tomorrow, I have generated multiple backgrounds for each class, the result on yolov3 is extremely good:) thank you

[anguoyang]

@AlexeyAB I have just tested the results on reinforcement, it is seems that there is no obvious improvements, I will try to train more iters and test more images.
What impact the result most is the data generation, I have generate more images with different background, multi-class in an image and over-lapping, in a word, the data is more similar with the testing scenario, the result become more better on yolov3.

[anguoyang]

for example, I need to detect objects holding in a hand, then I will generate a hand on the object(crop the hand image into the object with some overlap, yes, the image then looks weird,haha ), and if detect the objects in a real scenario(different with the background of training data), the result is also much better.

[anguoyang]

the result of reinforcement as follow:
detections_count = 108068, unique_truth_count = 94797
class_id = 0, name = 2400003243950, ap = 100.00 %
class_id = 1, name = 4560236040097, ap = 100.00 %
class_id = 2, name = 4582229061032, ap = 100.00 %
class_id = 3, name = 4901601317121, ap = 90.91 %
class_id = 4, name = 4960833007309, ap = 100.00 %
class_id = 5, name = 4964408100037, ap = 100.00 %
class_id = 6, name = 4972444014249, ap = 100.00 %
class_id = 7, name = 4973221030827, ap = 100.00 %
class_id = 8, name = 4975759020523, ap = 100.00 %
class_id = 9, name = 4975759020608, ap = 99.86 %
class_id = 10, name = 4977564092849, ap = 100.00 %
class_id = 11, name = 4978955001846, ap = 99.95 %
class_id = 12, name = 4986449080666, ap = 100.00 %
class_id = 13, name = 4987241127054, ap = 99.99 %
class_id = 14, name = 4995770026502, ap = 100.00 %
for thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00
for thresh = 0.25, TP = 94782, FP = 323, FN = 15, average IoU = 91.56 %

mean average precision (mAP) = 0.993801, or 99.38 %
Total Detection Time: 1873.000000 Seconds

[AlexeyAB]

@anguoyang Thanks! Very good precision (mAP) = 0.993801, or 99.38 %
But what mAP can you get without reinforcement?

[anguoyang]

@AlexeyAB this is the mAP without reinforcement

[anguoyang]

as for the result:
./darknet detector map obj.data cfg/yolov3-obj.cfg backup/yolov3-obj_10000.weights -thresh 0.25
The obj.data is the same with training, so maybe I need to change to another dataset for testing?

[AlexeyAB]

@anguoyang Yes, you can use another dataset.
So what mAP with reinforcement?

[anguoyang]

@AlexeyAB I have not tested the result with reinforcement according to the above command, but tested on some images which is different with the training data(with the weights after training with reinforcement), the result is quite similar.
I will try to test the reinforcement according to the above command today

[anguoyang]

@AlexeyAB I have tested on the test dataset which has not been trained, the result as follow:

detections_count = 45261, unique_truth_count = 31469
class_id = 0, name = 2400003243950, ap = 98.96 %
class_id = 1, name = 4560236040097, ap = 100.00 %
class_id = 2, name = 4582229061032, ap = 90.91 %
class_id = 3, name = 4901601317121, ap = 100.00 %
class_id = 4, name = 4960833007309, ap = 99.72 %
class_id = 5, name = 4964408100037, ap = 100.00 %
class_id = 6, name = 4972444014249, ap = 90.91 %
class_id = 7, name = 4973221030827, ap = 100.00 %
class_id = 8, name = 4975759020523, ap = 99.99 %
class_id = 9, name = 4975759020608, ap = 99.72 %
class_id = 10, name = 4977564092849, ap = 99.55 %
class_id = 11, name = 4978955001846, ap = 90.91 %
class_id = 12, name = 4986449080666, ap = 100.00 %
class_id = 13, name = 4987241127054, ap = 90.91 %
class_id = 14, name = 4995770026502, ap = 99.60 %
for thresh = 0.25, precision = 0.99, recall = 1.00, F1-score = 0.99
for thresh = 0.25, TP = 31398, FP = 365, FN = 71, average IoU = 87.74 %

mean average precision (mAP) = 0.974119, or 97.41 %

Total Detection Time: 549.000000 Seconds

it is a little bit lower than on training dataset, but it also acceptable

[AlexeyAB]

Thanks!

[anguoyang]

@AlexeyAB I have another question, would you please give me some advices? thank you.
As far as I know, the pretrained model of V3 has 80 classes, including car and person, if I train my custom objects, is it possible to keep the detection on car and person? because the obj.data includes only the custom objects, as well as the obj.names, how should I do if I want to detect custom objects and person/car, do I need to re-train with custom dataset and also COCO car/person related dataset?

[AlexeyAB]

@anguoyang You should train from the begining. You can use Dataset that contains New images with your object + COCO images with car+preson.
But check, that if your object, for example, cat, that there aren't non-labeled cats in the COCO-images.

[anguoyang]

@AlexeyAB ok, thank you