Darknet: Continue training or train from begining, with modified/increased training/val images ...

[VanitarNordic]

Hello,

I've heard if we gather those images which a model finds false positive objects inside, then annotate and add them inside existing images dataset, then we can increase the accuracy of the model dramatically.

in this case, assume that we have trained the model for 2000 iterations and we have a weight file of this. Then we add these new images into the dataset.

Should we start training from the beginning (from 1) or we can continue training by 2001?

[AlexeyAB]

@VanitarNordic

It is better to train from the beginning. But not necessarily.

• If you will continue to train from 2001 iteration only with dataset of new 500 images, then the network will gradually learn new images, but gradually forget old images.

• If you have 2000 images and has trained 2000 iterations, then find new images with false positive or false negative detections, for example 500 images, then you can add it to first dataset and continue training from 2000 iterations with this 2500 images, but too many iterations can lead to overfitting, i.e. you will can to detect only images from the dataset: https://en.wikipedia.org/wiki/Overfitting

Example of overfitting:

You should necessarily to train from the beginning only if your dataset is wrong, or if you change your Network .cfg-file, for example, if you change number of objects (classes).

[VanitarNordic]

Yess, I am agree with you, from the beginning is better, at least to avoid over-fitting as you mentioned correctly. Brilliant. Please check the YOLO_Mark also, I have posted a small issue there.

[VanitarNordic]

@AlexeyAB

is there anyway to verify the validation set errors during training? I mean which parameter in the darknet Yolo-2 training belongs to that?

avg error seems to be for the Training set.

[AlexeyAB]

@VanitarNordic

No. To validate, you should stop training, get one of weights, and do the validation:

• put path-names of validation images to file data\voc.2007.test (or simply copy data\train.txt to data\voc.2007.test)
• darknet.exe detector recall data/obj.data yolo-obj.cfg yolo-obj_9000.weights -i 0

You will get such result, where last line is main:

7580 7606 7683 RPs/Img: 68.21 IOU: 77.86% Recall:99.00%
7581 7607 7684 RPs/Img: 68.21 IOU: 77.86% Recall:99.00%
7582 7608 7685 RPs/Img: 68.21 IOU: 77.86% Recall:99.00%
7583 7609 7686 RPs/Img: 68.22 IOU: 77.86% Recall:99.00%
7584 7610 7687 RPs/Img: 68.22 IOU: 77.86% Recall:99.00%
7585 7611 7688 RPs/Img: 68.22 IOU: 77.86% Recall:99.00%
7586 7612 7689 RPs/Img: 68.23 IOU: 77.86% Recall:99.00%

D:Darknet2darknet\builddarknetx64>pause

[VanitarNordic]

it worked, in my case:

By the way, What are the RPs/Img and IOU values and we should try to make them smaller or bigger?

[AlexeyAB]

RPs/Img, IOU and Recall calculated here: https://github.com/AlexeyAB/darknet/blob/2a9a5229c87c5e05c87d9d792c62cf020b3f1981/src/detector.c#L432

• RPs/Img - average found objects per image (nothing say about accuracy)
• IOU - the bigger, the better (says about accuracy) - better to use
• Recall - the bigger, the better (says about accuracy)

[VanitarNordic]

How many training and validation images are typically needed to make a good detector?

because in my case, I have made a one class apple detector model with a small dataset, although calculations shows very good avg error and IOU, but the model still detects many false-positives.

[AlexeyAB]

@VanitarNordic

• training images required about 2000 per class (object)
• validation images required about 500 per class (object)

I added some manual When should I stop training: https://github.com/AlexeyAB/darknet#when-should-i-stop-training

You can put here some of image from training data-set and false-positives results, as example, may be I'll can say something about it.

[VanitarNordic]

this is an example of a false detection. it has detected the back of a monitor as apple. anyway dataset is very small, it is natural maybe (130/25 training/validation images). I make these for experiment

[AlexeyAB]

Try to add all images with false-positive to training dataset.

[VanitarNordic]

Try to add all images with false-positive to training dataset.

Yes, that's correct. I am agree.

Also, one question in side, Does amount of computer RAM (not the GPU memory) affects the training?

When I train, the computer becomes unstable and slow (once dropped of the training in the middle of the job around few hundreds of iterations, I did not see the error and reason of that).

I have 8G DDR4-2400 RAM installed, I'm curious if increasing it to 16G would solve the case, specially in future when I want to train bigger datasets.

the image shows consumption of resources, considering Windows 10 itself consumes around 2-3G of RAM, should I increase it?

[VanitarNordic]

another interesting point is when I process a single image, it says it is predicted in 0.024 second. It means that I should get around 41 FPS in real time live video, but in practice I get variations between 8 to 20FPS, why?!

The number is correct when I process a video file, close to calculation, 38 to 40FPS

[AlexeyAB]

• Increase amount of DDR CPU RAM don't increase performance of training, but it can help to avoid errors Out of memory. Especially if at this point that something else running on the computer.

• To process a single image measured time only for network_predict(net, X);
  But to proccess a video measured the highest time of:

  1. network_predict(net, X); get_region_boxes(); do_nms(); draw_detections(); ... in detect_in_thread
  2. in = get_image_from_stream(cap); in fetch_in_thread
  3. show_image(disp, "Demo"); which also save video to mpeg-file cvWriteFrame(output_video, disp);:

I.e. for video, there are also executed many other functions, not only network_predict(net, X);: https://github.com/AlexeyAB/darknet/blob/2a9a5229c87c5e05c87d9d792c62cf020b3f1981/src/demo.c#L53

another interesting point is when I process a single image, it says it is predicted in 0.024 second. It means that I should get around 41 FPS in real time but in practice I get variations between 8 to 20FPS, why?!

[VanitarNordic]

The number is correct when I process a video file, close to calculations, 38 to 40FPS, issue only happens with network camera

[AlexeyAB]

Network camera can send only that numbers of frames, which you set on it :)

[VanitarNordic]

hahaha, yes it solved but it varies too much. for a video file it is more stable. for the network camera it varies from 5 to 39FPS.

The only two issues remained here at the moment one is detection of small objects which I have problem with and also making some GUI application based on your repository.

[AlexeyAB]

About small object, there is solution - 1.2. If you want to detect small object on images with 832x832 or 1088x108: https://github.com/AlexeyAB/darknet/issues/12#issuecomment-283340030

[VanitarNordic]

That was a weird error which I told you above. Now I catch it and got a picture. The reason is unknown.

[AlexeyAB]

Did you do on your computer for something else at the same time?
I think something else took the GPU-RAM or the GPU-RAM.

[VanitarNordic]

no nothing, I close all programs when I decide to train. besides as you see in the picture in some comments above, only a half of GPU was occupied.

Besides, computer becomes slow and sometimes mouse pointer has difficulties in movement but it does not freeze. This phenomenon intensifies if I use high res training images. It does not affect GPU memory consumption but it affects the system performance.