Javacv: FFmpegFrameRecorder memery issue

[Allenisme]

private void writePacket(int mediaType, AVPacket avPacket) throws Exception {

        AVStream avStream = (mediaType == AVMEDIA_TYPE_VIDEO) ? audio_st : (mediaType == AVMEDIA_TYPE_AUDIO) ? video_st : null;
        String mediaTypeStr = (mediaType == AVMEDIA_TYPE_VIDEO) ? "video" : (mediaType == AVMEDIA_TYPE_AUDIO) ? "audio" : "unsupported media stream type";

        synchronized (oc) {
            int ret;
            if (interleaved && avStream != null) {
                if ((ret = av_interleaved_write_frame(oc, avPacket)) < 0) {
                    throw new Exception("av_interleaved_write_frame() error " + ret + " while writing interleaved " + mediaTypeStr + " packet.");
                }
            } else {
                if ((ret = av_write_frame(oc, avPacket)) < 0) {
                    throw new Exception("av_write_frame() error " + ret + " while writing " + mediaTypeStr + " packet.");
                }
            }
        }
    }

In the end of the "writePacket" method, it does not free the avPacket. So is there a way to do that or should we insert "avcodec.av_packet_unref(avPacket)" statement?

[saudet]

The packet data is initialized with buffers allocated independently, so I don't think we need to call av_packet_unref(), but if you think something is missing, please try it out, and let me know the results. Thanks!

[Allenisme]

I tried with such codes:
while ((frame = grabber.grab()) != null) { videoTs = 1000 * (System.currentTimeMillis() - startTime); if (videoTs > recorder.getTimestamp()) { logger.info(" -- Lip-flap correction: " + videoTs + " : " + recorder.getTimestamp() + " -> " + (videoTs - recorder.getTimestamp()) + " -- "); recorder.setTimestamp(videoTs); } recorder.record(frame); }
Without calling av_packet_unref(), the memery kept increasing. I have read the source code of FFmpegFrameRecorder, and i saw av_free_packet(pkt); statement in recordPacket(AVPacket pkt) method. But I didn't see any statement to free packets in recordImage(); method.

[saudet]

So, could you please try it if you think it's missing?

[Allenisme]

I tried, and the memery problem is improved, but it's still keep increasing at a very low speed.

[saudet]

Ok, that's an improvement, could you please send a pull request?

[saudet]

@mifritscher is also having similar issues. If you could help each other debug this, that would be great!

@Allenisme, thanks for finding one leak already!

[mifritscher]

hmm, if I unterstand av_init_packet(video_pkt); correctly, it does init the members of video_pkt. I'm wondering if av_free_packet is the right way, but could it be that buffers which are referenzed by the AVPacket dont get freed?

[mifritscher]

Moving av_free_packet(avPacket) from recordPacket to writePacket indeed helps - in my case at least to 99,9% - for the other 0,1% I would see it only after a day or so ;-)

Thanks!

[kellyho9021]

I am having out of memory issue after running FFmpegFrameGrabber for about 1 hour. I tried 1.4.4 SNAPSHOT version but still experienced the same issue. Below is the exception details that I got from Netbeans's profiler.

"Video Processing Thread" prio=1 tid=64 RUNNABLE
at java.lang.OutOfMemoryError.(OutOfMemoryError.java:48)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:233)
Local Variable: byte[]#6525
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
Local Variable: java.io.BufferedInputStream#3
at org.bytedeco.javacv.FFmpegFrameGrabber$ReadCallback.call(FFmpegFrameGrabber.java:260)
Local Variable: byte[]#6527
Local Variable: org.bytedeco.javacpp.BytePointer#2
at org.bytedeco.javacpp.avformat.av_read_frame(Native Method)
at org.bytedeco.javacv.FFmpegFrameGrabber.grabFrame(FFmpegFrameGrabber.java:1146)
at org.bytedeco.javacv.FFmpegFrameGrabber.grab(FFmpegFrameGrabber.java:1097)
at org.bytedeco.javacv.FrameGrabber.grabFrame(FrameGrabber.java:539)
at com.srcinc.commongui.rtsvf.processors.in.VideoProcessor$ProcessVideoAction.run(VideoProcessor.java:310)
Local Variable: com.srcinc.commongui.rtsvf.processors.in.VideoProcessor$ProcessVideoAction#1
Local Variable: org.bytedeco.javacpp.PointerScope#1
Local Variable: com.srcinc.commongui.rtsvf.data.impl.BaseVideoFeed#1
Local Variable: org.bytedeco.javacv.FFmpegFrameGrabber#1
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Local Variable: java.util.concurrent.Executors$RunnableAdapter#6
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#6
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#8
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#9
at java.lang.Thread.run(Thread.java:748)

[saudet]

InputStream doesn't support seeking, but FFmpeg requires it, so the whole
stream gets cached in memory. If you have a stream that doesn't fit in
memory, you'll probably need to find a way that doesn't involve InputStream.

[kellyho9021]

I just tried to use older version of JavaCV and it seemed to work fine. I didn’t get that exception after an hour of playing like when I used 1.4.4 snapshot and 1.4.3. The heap memory showed in profiler seemed to be more efficient as well

[saudet]

Some streams can work without seeking, so we could disable caching in those
case, true. Anyway, this isn't related to this issue, so please open
another issue, or even better, a pull request with the fix! Thanks

[saudet]

@kellyho9021 Actually, it's a pretty simple matter. I've just fixed this in commit https://github.com/bytedeco/javacv/commit/6ae006a82745ad7e84914581a44ffbe1dc82ee0a.
Please give it a try with 1.4.4-SNAPSHOT! http://bytedeco.org/builds/

[kellyho9021]

@saudet Thanks for the help.
I tried latest version of 1.4.4-SNAPSHOT but still got the same issue. The issue occurred after 40 mins of running.

This appears in the log
2019-01-10 10:06:33.605 [StdErr Flush] INFO stderr - Error on InputStream.read(): java.lang.OutOfMemoryError: Java heap space

This appears in Netbean's HeapWalker
"Video Processing Thread" prio=1 tid=64 RUNNABLE
at java.lang.OutOfMemoryError.(OutOfMemoryError.java:48)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:233)
Local Variable: byte[]#5579
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
Local Variable: java.io.BufferedInputStream#3
at org.bytedeco.javacv.FFmpegFrameGrabber$ReadCallback.call(FFmpegFrameGrabber.java:266)
Local Variable: byte[]#5584
Local Variable: org.bytedeco.javacpp.BytePointer#2
at org.bytedeco.javacpp.avformat.av_read_frame(Native Method)
at org.bytedeco.javacv.FFmpegFrameGrabber.grabFrame(FFmpegFrameGrabber.java:1153)
at org.bytedeco.javacv.FFmpegFrameGrabber.grabImage(FFmpegFrameGrabber.java:1107)
at com.srcinc.commongui.rtsvf.processors.in.VideoProcessor$ProcessVideoAction.run(VideoProcessor.java:272)
Local Variable: org.bytedeco.javacv.FFmpegFrameGrabber#1
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Local Variable: com.srcinc.commongui.rtsvf.processors.in.VideoProcessor$ProcessVideoAction#1
Local Variable: java.util.concurrent.ThreadPoolExecutor#8
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#9
at java.lang.Thread.run(Thread.java:748)

[saudet]

You're going to need to set the maximumSize to something a lot lower than the default.

[kellyho9021]

Thanks @saudet. I didn't change the maximumSize on that test. I'm testing it again. There's no issue so far

[saudet]

The fixes are now included in JavaCV 1.4.4. Thanks for the contribution and for all the testing! Please let me know if other memory leaks show up.

[saudet]

BTW, I think I've fixed all the memory leaks occurring in FFmpegFrameGrabber and FFmpegFrameRecorder.
Please give it a try with the snapshots: http://bytedeco.org/builds/