Darknet: Time shifting of "res.avi"

[VanitarNordic]

Hello,

One method which I use to enhance the model, is that I re-enter the false positive images as background-only into training dataset. majority of times the input source is a video file and I should use snapshots of the video to make images of the scenes.

Therefore I compare the res.avi with source video to see at which time I face the false positives. Then I take snapshot of that scene from the source file, because the snapshot from the res.avi can not be used (because it prints bounding boxes on scenes which alter the models behavior, if I use them as source)

Alright, now the problem is that the res.avi and source video do not have the same timing in many scenes. for example a scene at 5,123 in res.avi is not identical with same time scene in the source video file. They are different.

Do you have any suggestion to overcome this?

[AlexeyAB]

Hi,

• You can set FRAME 1 and use darknet.sln: https://github.com/AlexeyAB/darknet/blob/9e23ff2523713123daea5ff78fa93737cae349c6/src/demo.c#L18

• Or use build\darknet\yolo_console_dll.sln - there each frame should be identical in source and result video, just add these 2 lines here before draw_boxes(): https://github.com/AlexeyAB/darknet/blob/9e23ff2523713123daea5ff78fa93737cae349c6/src/yolo_console_dll.cpp#L104

static cv::VideoWriter output_video("result.avi", CV_FOURCC('D', 'I', 'V', 'X'), cap.get(CV_CAP_PROP_FPS), prev_frame.size(), true);
output_video << prev_frame;

[VanitarNordic]

You can set FRAME 1 and use darknet.sln

I used this but still the results are not identical. Do you have more suggestions on this or you prefer that I try the second method?

[VanitarNordic]

and one thing,

is it possible to make the result output to a format rather than .avi? for example .mp4

I use the VLC for this purpose, because it only handles AVI files with this resolution. I don't want to convert it to other formats with other software, but I'm curious if it's possible to publish in MP4 with Darknet itself. Then, I can check this timing with other software also rather than VLC. Maybe VLC itself also has some problems.

[AlexeyAB]

• Also comment these 3 lines: https://github.com/AlexeyAB/darknet/blob/9e23ff2523713123daea5ff78fa93737cae349c6/src/demo.c#L79

    memcpy(predictions[demo_index], prediction, l.outputs*sizeof(float));
    mean_arrays(predictions, FRAMES, l.outputs, avg);
    l.output = avg;

• And comment these 4 linse: https://github.com/AlexeyAB/darknet/blob/9e23ff2523713123daea5ff78fa93737cae349c6/src/demo.c#L97

    images[demo_index] = det;
    det = images[(demo_index + FRAMES/2 + 1)%FRAMES];
    ipl_images[demo_index] = det_img;
    det_img = ipl_images[(demo_index + FRAMES / 2 + 1) % FRAMES];

• Or try to use build\darknet\yolo_console_dll.sln

---

Just try to use: darknet.exe detector demo data/voc.data yolo-voc.cfg yolo-voc.weights test.mp4 -i 0 -out_filename res.mp4

• use any container: avi or mpeg https://github.com/AlexeyAB/darknet/blob/master/build/darknet/x64/darknet_demo_store.cmd#L1

• use any codec: divx, h264, mjpg https://github.com/AlexeyAB/darknet/blob/9e23ff2523713123daea5ff78fa93737cae349c6/src/image.c#L544

[VanitarNordic]

Just try to use: darknet.exe detector demo data/voc.data yolo-voc.cfg yolo-voc.weights test.mp4 -i 0 -out_filename res.mp4

Using this command does not create res.mp4. I get these errors. I'll try to change the codec.

[VanitarNordic]

I changed to H264, I get this error:
Could not find encoder for codec id 28: Encoder not found

changed to mjpg, I get this error:
[mp4 @ 000001f721da36c0] Tag MJPG/0x47504a4d incompatible with output codec id '8' (l[0][0][0])

[AlexeyAB]

Yes, there is problem to save to mp4 using OpenCV: https://www.google.ru/search?q=OpenCV+VideoWriter+mp4&rlz=1C1MSIM_enRU714RU714&oq=OpenCV+VideoWriter+mp4&aqs=chrome..69i57j69i60l3.3736j0j7&sourceid=chrome&ie=UTF-8

But you can save to .avi or .mpeg using codec CV_FOURCC('D', 'I', 'V', 'X') it can be opened using: Windows Media Player, VLC, Power DVD.

Also you can try to use other codecs:

output_video = cvCreateVideoWriter(out_filename, CV_FOURCC('H', '2', '6', '4'), 25, size, 1);
output_video = cvCreateVideoWriter(out_filename, CV_FOURCC('D', 'I', 'V', 'X'), 25, size, 1);
output_video = cvCreateVideoWriter(out_filename, CV_FOURCC('M', 'J', 'P', 'G'), 25, size, 1);
output_video = cvCreateVideoWriter(out_filename, CV_FOURCC('M', 'P', '4', 'V'), 25, size, 1);
output_video = cvCreateVideoWriter(out_filename, CV_FOURCC('M', 'P', '4', '2'), 25, size, 1);
output_video = cvCreateVideoWriter(out_filename, CV_FOURCC('X', 'V', 'I', 'D'), 25, size, 1);
output_video = cvCreateVideoWriter(out_filename, CV_FOURCC('W', 'M', 'V', '2'), 25, size, 1);

---

Also you can try to install: https://github.com/cisco/openh264/releases
and use CV_FOURCC('X','2','6','4') to save mp4
http://answers.opencv.org/question/95238/how-to-save-a-video-in-mp4-format/?answer=95480#post-id-95480

[VanitarNordic]

Thank you.

Also you can try to install: https://github.com/cisco/openh264/releases
and use CV_FOURCC('X','2','6','4') to save mp4
http://answers.opencv.org/question/95238/how-to-save-a-video-in-mp4-format/?answer=95480#post-id-95480

I did this. there is a file named openh264-1.7.0-win64.dll. I put it nearDarknet.exe and recompiled and tested but again I get this error
Could not find encoder for codec id 28: Encoder not found
and no output mp4 file.

[VanitarNordic]

But you can save to .avi or .mpeg using codec CV_FOURCC('D', 'I', 'V', 'X') it can be opened using: Windows Media Player, VLC, Power DVD.

I tried res.mpeg, but the result file can not be played by no player. file has a capacity, but can not be played.

[AlexeyAB]

But you can save to .avi or .mpeg using codec CV_FOURCC('D', 'I', 'V', 'X') it can be opened using: Windows Media Player, VLC, Power DVD.

I tried res.mpeg, but the result file can not be played by no player. file has a capacity, but can not be played.

I just right now did it and it works well, I can see res.mpeg video using Windows Media Player, VLC and Power DVD.

[VanitarNordic]

I did it now, but I could not play it. Do you have some codec installed?

[AlexeyAB]

No

[VanitarNordic]

Also comment these 3 lines:
darknet/src/demo.c
Line 79 in 9e23ff2
memcpy(predictions[demo_index], prediction, l.outputssizeof(float));
memcpy(predictions[demo_index], prediction, l.outputssizeof(float));
mean_arrays(predictions, FRAMES, l.outputs, avg);
l.output = avg;
And comment these 4 linse:
darknet/src/demo.c
Line 97 in 9e23ff2
images[demo_index] = det;
images[demo_index] = det;
det = images[(demo_index + FRAMES/2 + 1)%FRAMES];
ipl_images[demo_index] = det_img;
det_img = ipl_images[(demo_index + FRAMES / 2 + 1) % FRAMES];

I tried these also with FRAME 1
but still the frames or timing does not match. You can test yourself. I use a plugin named Jump to time 2.1. it can be installed from VLC plugins also.

[VanitarNordic]

Okay, I found why this happens.

Actually the FPS in the source file and res.avi must be identical, otherwise they will not match.
These are the required steps:

1) #define FRAMES 1
2) changing the FPS parameter in the code (default is 25) to match the source video file's FPS
output_video = cvCreateVideoWriter(out_filename, CV_FOURCC('D', 'I', 'V', 'X'), 25, size, 1);
or record a video or use a video with FPS=25 to match the FPS in the code.