Darknet: Significative difference between detector demo and YOLO C++ API fps

[angeligareta]

I am having a significative difference detecting elements from the same video footage in two different ways:

• Running ./darknet detector demo... I achieve a rate of 120 fps in the video.
• Using QT and Yolo C++ API I get 60-70 fps.

Could anyone tell me why is this difference? Do you think it would be better to use darknet by command line from QT?

Thanks in advance!

[AlexeyAB]

How many FPS do you get with yolo_console_dll.cpp
LD_LIBRARY_PATH=./:$LD_LIBRARY_PATH ./uselib data/coco.names cfg/yolov3.cfg yolov3.weights test.mp4
without Qt and without your modifications?

[angeligareta]

I have just tried it and I get 120 fps too... Is it only for using QT?

The way I use the Yolo C++ API is:

void YoloDetector::processVideoFile(const std::vector<std::string>& element_names)
{
    cv::Mat frame;
    yolo_detector_->nms = NMS_THRESHOLD; // Set nont-maximum supression threshold for track_id

    unsigned int frame_counter = 1;
    cv::VideoCapture cap(getInputFile());
    while(!hasExitSignal() && cap.isOpened()) {
        // Read new capture
        cap >> frame;

        startTimer();
        // Detect and track
        std::vector<bbox_t> results = yolo_detector_->detect(frame, DETECTION_THRESHOLD);
        results = yolo_detector_->tracking_id(results);
        stopTimer();

       // Get difference between end time and start time
       detection_window_ -> displayFPS("AVERAGE FPS: " + std::to_string(getCurrentFPS()) + " fps");

        // Draw boxes and display image.
        try {
            drawBoxes(frame, results, element_names);
            detection_window_ -> displayMainImage(frame);
        } catch (...) {}

        // For not blocking UI
        QCoreApplication::processEvents();
        frame_counter ++;
    }

    cap.release();
}

[AlexeyAB]

I have just tried it and I get 120 fps too... Is it only for using QT?

No.
It is because you use only 1 CPU-thread. You should use 3 threads

1. Capture
2. Detection
3. Drawing, showing, saving

As it is done in yolo_console_dll.cpp

[angeligareta]

Thanks for the help @AlexeyAB! I will try to implement it using 3 threads!

Btw, I developed Datasets2Darknet a modular tool that extracts images and labels from multiple datasets and parses them to Darknet format, maybe it would be helpful to others if you add it in your README.md.

[AlexeyAB]

@angeligareta Thanks! Can it be used to convert labels from: MS COCO 2018, Kitty, Cityscapes, ImageNet VID, BDD100K, Open Images, Market-1501... to Darknet labels? Or other datasets: https://github.com/AlexeyAB/darknet/tree/master/scripts#datasets

Can you show simple command line how to do this?

[angeligareta]

Thanks for the interest @AlexeyAB!

The aim of the Datasets2Darknet is to combine several datasets and parsing the annotations and images to a common format and centralized location. For parsing a new dataset a specific parser for that dataset must be created for indicating where to take the data from.

For the moment, I have implemented dataset parsers, located in Darknet/src/datasets_parsers for:

• German Traffic Sign Recognition Benchmark: http://benchmark.ini.rub.de/
• German Traffic Sign Detection Benchmark (classification): http://benchmark.ini.rub.de/?section=gtsdb
• Belgium Traffic Sign Dataset: https://btsd.ethz.ch/shareddata/
• Mapping and Assessing the State of Traffic InFrastructure (MASTIF) Dataset: http://www.zemris.fer.hr/~ssegvic/mastif/datasets.shtml
• LISA Traffic Sign Dataset: http://cvrr.ucsd.edu/vivachallenge/index.php/signs/sign-detection/
• LISA Traffic Light Dataset: http://cvrr.ucsd.edu/LISA/lisa-traffic-sign-dataset.html
• Russian Traffic Sign Dataset: http://graphics.cs.msu.ru/en/research/projects/rtsd

The idea is to share the project in your fork for the people to be able to contribute and create specific parsers for more datasets! Everything is well documented in the README.md and I have added a tutorial on how to create a new parser.

I think the idea is really good, what is your opinion?

[AlexeyAB]

@angeligareta Hi,

Yes, I think it is a good idea. Just add step-by-step tutorial for converting one of datasets, for users who don't know python. So I will add URL to your repository in Readme.md
For example, to which folder should I put images and to which folder should I put labels? - for each Dataset: German Traffic Sign, Belgium Traffic Sign, ...

Does it convert only Classification(Recognition) labels, or also Detection labels?

[angeligareta]

Hi @AlexeyAB

I have added instructions to Datasets2Darknet for using the available datasets, creating a new dataset parser and a list of the currently available datasets.

I hope you like it and you can share it with your users. Btw, could I do a pull request with new Python utilities, I have created four new Python programs that can be very useful. I would love to be a contributor to this amazing project!

• Python Program for calculating the mAP for darknet classifier (not implemented before).
• Example of classifying an image with Python API.
• Python Program to calculate the mAP for multiple detection weights at the same time and rank them according to their precision (very useful to decide which weight is better in your backup folder).
• Python Program that calculates the average precision for each class of a darknet classifier model weight

[AlexeyAB]

@angeligareta Hi, Thanks!

1. Can you add ability to specify all required variables through specifying command line parameters instead of changing Python-source code in Datasets2Darknet?

Example of classifying an image with Python API.

Yes, I think it is a good idea to have Python script for classifier. Just may be it should be in subdir /scripts/

---

Python Program for calculating the mAP for darknet classifier (not implemented before).

1. What is the mAP for classifier? For Classifier there can be only TopK and ErrorK=(100-TopK), while the mAP can be applied only for Detector: https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173

There are Top1 and Top5 for classifier:
darknet.exe classifier valid cfg/imagenet1k.data cfg/darknet53_448_xnor.cfg backup/darknet53_448_xnor_last.weights

And we can see TopK (K is specified in data-file) during training
darknet.exe classifier train cfg/imagenet1k.data cfg/darknet53_448_xnor.cfg darknet53_xnor.conv.74 -topk

---

Python Program to calculate the mAP for multiple detection weights at the same time and rank them according to their precision (very useful to decide which weight is better in your backup folder).

For example, what is the advantage of this script over getting the _best.weights that is automatically created during training if -map flag is spesified?

[angeligareta]

HI @AlexeyAB,

I have just finished modifying Datasets2Darknet, I have added click command line arguments as you recommended me, nice advice! And now there are several available options such as root_path, train_pct, test_pct and so on. You can see more information executing in Datasets2Darknet root.

python3 src/general_parser.py --help

In respect of the Python Program for calculating the mAP for darknet classifier (not implemented before), it calculates the precision for each class in the names list and the average precision for those weights, so it is far better than the actual .\darknet classifier valid, that only shows the topK precision for each image in the validation list.

In the Python Program to calculate the mAP for multiple detection weights at the same time and rank them according to their precision (very useful to decide which weight is better in your backup folder). you can select how many top weights to keep, not only the best one but for example, a top 5. But yes, it is not as useful as the others.

Tell me what to do for accepting the pull request and I will do it! I am so happy to be able to contribute to this project! Regards

[angeligareta]

Hi @AlexeyAB,

So will you add Datasets2Darknet to your README.md or you need anything else?

Regards!

[AlexeyAB]

@angeligareta Hi,
Yes, I added Datasets2Darknet URL to readme.md: https://github.com/AlexeyAB/darknet#datasets

And now there are several available options such as root_path, train_pct, test_pct and so on.

Can you add examples and descriptions of these parameters to your Readme.md?

Also can you fix some issue? You specified that in Detection Task https://github.com/angeligareta/Datasets2Darknet#detection-task there is German Traffic Sign Recognition Benchmark, but Recognition is Classifier, not Detector, since there are two sets: http://benchmark.ini.rub.de/

• German Traffic Sign Recognition Benchmark (GTSRB), a large multi-category classification benchmark,
• and the German Traffic Sign Detection Benchmark (GTSDB).

[angeligareta]

Hi @AlexeyAB,

Thank you! I have just modified the second error. Tomorrow I will extend the README.md

Is there any possibility of adding me as a contributor to the project? And finally, do you want me to do a pull request with the Python programs?

Regards!

[angeligareta]

Hi @AlexeyAB

So what do I have to do for doing the pull request with the Python Programs?

Regards!