Javacv: Viode is not smooth with 1080p video resolution

[harischandraprasad]

Hi,

Using JavaCV 1.3.3.
I am writing an application to capture the video stream from IPCamera. I have written a small sample code by following sample code as mentioned in https://github.com/bytedeco/javacv.

My source of video stream is IPCamera with 25fps. Its working perfectly fine with 704x576, video was very smooth and almost no delay(less than 200ms). However, if I use video with 1080p(1920x1080) resolution delay seems fine as its around 600ms, however the video was not smooth as looks like the processing time to process each frame is very high. I just used the below code snippet to see the delay:

final String ffmpegSrcLink =  "rtsp://192.168.1.161:554/av0_0";
FrameGrabber grabber = new FFmpegFrameGrabber(ffmpegSrcLink);
grabber.start();
OpenCVFrameConverter.ToIplImage converter = new OpenCVFrameConverter.ToIplImage();
        IplImage grabbedImage = converter.convert(grabber.grab());

CanvasFrame frame = new CanvasFrame("Video Capture Test", CanvasFrame.getDefaultGamma()/grabber.getGamma());

long start = 0, end = 0;
        long prevTimeStamp = 0;
        long curTimeStamp;
        long totalFrameCount = 0;
        String msg = "";
while (frame.isVisible() && (grabbedFrame = grabber.grab()) != null) {
            curTimeStamp = grabber.getTimestamp();
            msg = String.format("\nGot a frame(%3d): (%4d) - (%7d)",
                totalFrameCount,
                (System.currentTimeMillis() - start),
                (curTimeStamp - prevTimeStamp));
            System.out.println(msg);
            prevTimeStamp = curTimeStamp;
            start = System.currentTimeMillis();

            grabbedImage = converter.convert(grabbedFrame);
            frame.showImage(grabbedFrame);
            ++totalFrameCount;
        }
frame.dispose();
grabber.stop();

The value that I am checking is (curTimeStamp - prevTimeStamp). For 25fps, initially for few frames, its showing as 40,000. After that its taking too much time. Almost every alternate frame taking around 4 or 6 times and hence the video is not looking smooth. I have attached the two log files (out_1920x1080_high_res.log and out_704x576_low_res.log) that I captured from the console output with the above sample code.

Ex:(from out_1920x1080_high_res.log)
....
Got a frame( 5): ( 39) - ( 40000)
Got a frame( 6): ( 270) - ( 160000)
Got a frame( 7): ( 158) - ( 201000)
Got a frame( 8): ( 36) - ( 40000)
Got a frame( 9): ( 331) - ( 160000)
Got a frame( 10): ( 36) - ( 40000)
Got a frame( 11): ( 153) - ( 280000)
....
Where as with low res(out_704x576_low_res.log):
....
Got a frame( 10): ( 6) - ( 40000)
Got a frame( 11): ( 7) - ( 40000)
Got a frame( 12): ( 6) - ( 40000)
Got a frame( 13): ( 9) - ( 40000)
Got a frame( 14): ( 6) - ( 40000)
Got a frame( 15): ( 7) - ( 40000)
Got a frame( 16): ( 6) - ( 40000)
....

Is there any way I can improve this so that the frame processing time will not take very long time? any parameters that I can set to improve this?

Thanks for looking into it.

out_704x576_low_res.log
out_1920x1080_high_res.log

[saudet]

Can you do some profiling on the methods and show me the result?

[harischandraprasad]

Thanks for looking into it.

I ran the program using NetBeans and captured profile data and attached console out and profile data for both high and low resolutions. I am not sure whether the format is fine or not. If this is not the way that you are expecting, please let me know any other tool or command that I can use for capturing profile data.

profile_high_and_low_resolution.zip

[Updated ZIP file with .nps file for High Resolution]

[saudet]

Most of time is spent waiting in av_read_frame(), so this is just a bottleneck with your network. Please fix your network.

[harischandraprasad]

Thanks Samuel.

When I checked with _ffplay_ the video was smooth and much better than my sample app. I have tried like: _ffplay -fflags nobuffer rtsp://192.168.1.161:554/av0_0_.
I have tried even calling grabber.setOption("fflags", "nobuffer"); in my code, even then no difference. So, I thought something wrong in my code and did not suspected the issue with network. I will check about the network latency as well.

I have attached _ffplay_ console out with and without using _fflags=nobuffer_.
ffplay_with-and-without-flag-high_res.log

[saudet]

So, is your issue really about setting the "nobuffer" option?

[harischandraprasad]

I am guessing it might help, but not sure.
In fact, when I searched on net, I found some suggestion and I have tried all of the following options but no change:
grabber.setVideoOption("preset", "veryfast");
grabber.setVideoOption("fflags", "nobuffer");
grabber.setVideoOption("tune", "zerolatency");
grabber.setVideoOption("analyzeduration", "0");

I have tried with both setVideoOption() and setOption() as I am not sure which one to use.

[saudet]

Looking at the code, setOption("nobuffer", "1") is probably the correct way to set that option.

[harischandraprasad]

Thanks for looking into this.
Sorry for the delay, camera is not available to check this, so could not check until now.
Today I checked with setOption("nobuffer", "1") but no difference. Still I am getting delays, I will check latency on my network.

[saudet]

FYI, I've checked that again today and FFmpegFrameGrabber.setOption("fflags", "nobuffer") is probably the correct way to set that flag. Check the debug log to make sure it gets enabled.

[harischandraprasad]

Thanks a lot for keeping an eye on it and confirming on it. I will check it in debug logs.

Just for your information, I have implemented the same using ffmpeg iOS libraries and the behavior is exactly same. At the moment I am concentrating on the core functionality, so, I could not spend time on verifying network latency. Once I verify it I will definitely let you know.