Javacv: OutOfMemoryException by FFMpegFrameGrabber

Created on 13 Feb 2018  ·  50Comments  ·  Source: bytedeco/javacv

We are using FFMpegFrameGrabber for creating thumbnails out of videos.
We are facing OutOfMemoryException Like below.

Can someone help us??

. Thanks in advance.

HTTP Status 500 - Handler processing failed; nested exception is java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes = 3G > maxPhysicalBytes = 3G
type Exception report
message Handler processing failed; nested exception is java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes = 3G > maxPhysicalBytes = 3G
description The server encountered an internal error that prevented it from fulfilling this request.
exception
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes = 3G > maxPhysicalBytes = 3G
org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletionWithError(DispatcherServlet.java:1287)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
javax.servlet.http.HttpServlet.service(HttpServlet.java:644)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
org.openmrs.module.web.filter.ForcePasswordChangeFilter.doFilter(ForcePasswordChangeFilter.java:60)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:72)
org.openmrs.web.filter.GZIPFilter.doFilterInternal(GZIPFilter.java:64)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:70)
org.openmrs.module.webservices.rest.web.filter.AuthorizationFilter.doFilter(AuthorizationFilter.java:104)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:70)
org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:82)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:70)
org.openmrs.module.web.filter.ModuleFilter.doFilter(ModuleFilter.java:54)
org.openmrs.web.filter.OpenmrsFilter.doFilterInternal(OpenmrsFilter.java:108)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:150)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.openmrs.web.filter.StartupFilter.doFilter(StartupFilter.java:105)
org.openmrs.web.filter.StartupFilter.doFilter(StartupFilter.java:105)
org.openmrs.web.filter.StartupFilter.doFilter(StartupFilter.java:105)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
root cause
java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes = 3G > maxPhysicalBytes = 3G
org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:562)
org.bytedeco.javacpp.Pointer.init(Pointer.java:121)
org.bytedeco.javacpp.avcodec$AVPacket.allocate(Native Method)
org.bytedeco.javacpp.avcodec$AVPacket.(avcodec.java:1696)
org.bytedeco.javacv.FFmpegFrameGrabber.startUnsafe(FFmpegFrameGrabber.java:471)
org.bytedeco.javacv.FFmpegFrameGrabber.start(FFmpegFrameGrabber.java:462)
org.bahmni.module.bahmnimsf.extension.PatientDocumentThumbnailGenerator.generateThumbnail(PatientDocumentThumbnailGenerator.java:40)
org.bahmni.module.bahmnicore.service.impl.PatientDocumentServiceImpl.createAndSaveThumbnailForVideo(PatientDocumentServiceImpl.java:147)
org.bahmni.module.bahmnicore.service.impl.PatientDocumentServiceImpl.saveDocumentInFile(PatientDocumentServiceImpl.java:116)
org.bahmni.module.bahmnicore.service.impl.PatientDocumentServiceImpl.saveDocument(PatientDocumentServiceImpl.java:73)
org.bahmni.module.bahmnicore.web.v1_0.controller.VisitDocumentController.saveDocument(VisitDocumentController.java:60)
sun.reflect.GeneratedMethodAccessor1844.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:177)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:446)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:434)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
javax.servlet.http.HttpServlet.service(HttpServlet.java:644)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
org.openmrs.module.web.filter.ForcePasswordChangeFilter.doFilter(ForcePasswordChangeFilter.java:60)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:72)
org.openmrs.web.filter.GZIPFilter.doFilterInternal(GZIPFilter.java:64)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:70)
org.openmrs.module.webservices.rest.web.filter.AuthorizationFilter.doFilter(AuthorizationFilter.java:104)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:70)
org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:82)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:70)
org.openmrs.module.web.filter.ModuleFilter.doFilter(ModuleFilter.java:54)
org.openmrs.web.filter.OpenmrsFilter.doFilterInternal(OpenmrsFilter.java:108)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:150)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.openmrs.web.filter.StartupFilter.doFilter(StartupFilter.java:105)
org.openmrs.web.filter.StartupFilter.doFilter(StartupFilter.java:105)
org.openmrs.web.filter.StartupFilter.doFilter(StartupFilter.java:105)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.12 logs.
Apache Tomcat/8.0.12

bug question

All 50 comments

Others have noticed leaks as well. Would you have any profiler report to share?

I don't know how to see the profilers can you please help me with this?

This is the code we are using for creating thumbnail. If we are doing something wrong with the code you can suggest me. :)

If you are interested to see the code and debug. Please call me on skype. my id is [email protected]

  public BufferedImage generateThumbnail(File outputVideoFile) throws IOException {
    BufferedImage bufferedImage = null;
    try {
        FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(outputVideoFile);
        frameGrabber.start();
        Frame grabKeyFrame = frameGrabber.grabKeyFrame();
        Java2DFrameConverter java2DFrameConverter = new Java2DFrameConverter();
        bufferedImage = java2DFrameConverter.convert(grabKeyFrame);
        frameGrabber.stop();
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
    return bufferedImage;
}

Any problem with this perticular code. I am debugging it but not able found.

  java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes = 3G >  maxPhysicalBytes = 3G
 org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:562)
  org.bytedeco.javacpp.Pointer.init(Pointer.java:121)
  org.bytedeco.javacpp.avcodec$AVPacket.allocate(Native Method)
  org.bytedeco.javacpp.avcodec$AVPacket.<init>(avcodec.java:1696)
  org.bytedeco.javacv.FFmpegFrameGrabber.startUnsafe(FFmpegFrameGrabber.java:471)
  org.bytedeco.javacv.FFmpegFrameGrabber.start(FFmpegFrameGrabber.java:462)

Thanks for the code! It looks alright so, I guess there is a leak somewhere in native code, but someone is going to need to debug this... BTW, this is probably the same as issue #899.

Is it a closed issue? If it is close issue. which version we need to upgrade.

No, I've marked it as a duplicate issue and leaving this one open.

Do we have any quick fix for this?

If you don't mind using more memory, you can increase the threshold:
http://bytedeco.org/javacpp/apidocs/org/bytedeco/javacpp/Pointer.html#maxPhysicalBytes

Can I use this export _JAVA_OPTIONS="-Xms512m -Xmx1024m" command for increasing the physical memory?

Yes, that works too.

Thanks :) 👍

Hi, Small question/clarification how come export _JAVA_OPTIONS="-Xms512m -Xmx1024m" resolve the problem? can you explain?

On Windows, for example, native memory allocation/deallocation are expensive operations, especially when a lot of garbage has accumulated, such that JavaCPP gives up trying to free memory and aborts. By restricting the total amount of memory available, garbage collection is performed more frequently, with less delay. This might be what is happening here.

Oh thanks for the information :)

BTW, the new PointerScope feature in version 1.4.2 might help with that:
http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/

Hi, I'm suffering from the same OutOfMemory issue. I tried using PointerScope as you suggested, however a very strange behaviour happens. The code manages to read 1 file, then fails on consecutive files. I can easily reproduce the issue with just 2 files. Changing the order of the files doesn't affect the error. The file format I use is m4a/acc. If I remove the PointerScope, the code works fine, but memory grows indefinitely.

Here is the error:

avformat_open_input() error -2: Could not open input "java.io.BufferedInputStream@7ff35a3f". (Has setFormat() been called?)
org.bytedeco.javacv.FrameGrabber$Exception: avformat_open_input() error -2: Could not open input "java.io.BufferedInputStream@7ff35a3f". (Has setFormat() been called?)
    at org.bytedeco.javacv.FFmpegFrameGrabber.startUnsafe(FFmpegFrameGrabber.java:754)
    at org.bytedeco.javacv.FFmpegFrameGrabber.start(FFmpegFrameGrabber.java:690)

Here is my code:

Short[] _getAudioFloatArrayFromFile(InputStream fileStream, int targetSampleRate) throws FrameGrabber.Exception {
        try (PointerScope scope = new PointerScope()) {
            ArrayList<Short> ret = new ArrayList<>();
            FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(fileStream);
            grabber.setSampleFormat(1); // AV_SAMPLE_FMT_S16, signed 16 bits
            grabber.setSampleRate(targetSampleRate);
            grabber.setAudioChannels(1); // 1 channel mono audio
            grabber.setFormat("m4a");
            grabber.start();
            Frame frame = grabber.grabSamples();
            while (frame != null) {
                if (frame.samples == null) {
                    break;
                }

                ShortBuffer channelSamplesShortBuffer = (ShortBuffer) frame.samples[0];
                channelSamplesShortBuffer.rewind();

                for (int i = 0; i < channelSamplesShortBuffer.remaining(); i++) {
                    ret.add(channelSamplesShortBuffer.get(i));
                }
                frame = grabber.grabSamples();
            }
            grabber.stop();
            grabber.release();
            return ret.toArray(new Short[0]);
        }
    }

The callback objects are saved as static references, we'll need to add a null check for that...

@omarzouk I've fixed the issue with PointerScope in commit https://github.com/bytedeco/javacv/commit/e191d3f7a6c5f1c3effaf663b34da878efa184ac. Thanks for reporting!

Thanks a lot for the quick solution @saudet !

Since I need this urgently, I attempted to use 1.4.3-SNAPSHOT however due to the fact that the OSS Sonatype repo uses unique snapshot versions and I'm using SBT sa, i'm unable to resolve all the dependencies, the main reason being that in the metadata.xml of some libraries, the build number is different for different components which SBT doesn't really understand.

Any suggestions for how I can get around this?

Thanks!

We can use a dummy pom.xml file with Maven to download the artifacts, and
have sbt look in Maven's cache, similar to what is explained here:
https://deeplearning4j.org/android

@omarzouk If you're not able to get this working, just copy the new versions of source code itself of FFmpegFrameGrabber.java, etc to your project.

@saudet thanks! I just created a dummy pom with javacv-platform as dependency, and uploaded a fat jar with all dependencies to our company's internal nexus. This way SBT can just download the fat jar, and it will have everything.

I think PointerScope provides a good solution to this issue, and it works well with JavaCV 1.4.3 now. Thanks all for the feedback! If you're still having problems with PointerScope and the latest version of JavaCV though please let me know.

@saudet I am still facing the memory leak issue, when i am converting image to video using FFmpegFrameRecorder in while(true) loop

@gudduIgn You'll need to provide some details about what you're doing if you expect someone to help!

BTW, the new PointerScope feature in version 1.4.2 might help with that:
http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/

Hi saudet. How do I limit the maximum memory used by javacv? thanks

@duck123456 Set the "org.bytedeco.javacpp.maxphysicalbytes" system property:
http://bytedeco.org/javacpp/apidocs/org/bytedeco/javacpp/Pointer.html

@duck123456 Set the "org.bytedeco.javacpp.maxphysicalbytes" system property:
http://bytedeco.org/javacpp/apidocs/org/bytedeco/javacpp/Pointer.html

if Set the "org.bytedeco.javacpp.maxphysicalbytes" system property:,it will error occurred java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes (10990M) > maxPhysicalBytes (10923M)

It needs to set before calling anything else.

It needs to set before calling anything else.
class init use it! Is that a problem?

static{
System.setProperty("org.bytedeco.javacpp.maxPhysicalBytes","12G");
}

What does Pointer.maxPhysicalBytes() return?

What does Pointer.maxPhysicalBytes() return?

yes With the return

What does Pointer.maxPhysicalBytes() return?
There is no good way to close it, it will leak memory, Can you deal with it?
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L844

Can you send a pull request?

Both the buffer and the context are properly freed here, so I'm not sure what you want to fix:
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L248-L255

Both the buffer and the context are properly freed here, so I'm not sure what you want to fix:
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L248-L255

yes ,avio_alloc_context is no problem ,if use inputStream this no to close avformat_close_input
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L205-L208

it need to avformat_close_input(oc); Otherwise, memory leaks will occur

That's also all there:
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L256-L258

no,Just use it avformat_free_context still can't release it,you can try it

Ok, so please send a pull request with the proposed changes! I can't guess what you want to do here.

Ok, I'll test it again, and if it's ok, I'll pull request

I get repeatedly this error when loading high resolution (4K) and long videos. The execution hangs at the instruction FFmpegFrameGrabber.start() method.

2020-01-08 22:09:44.160 17361-17462/com.test.javacvapp W/mples.detectio: Throwing OutOfMemoryError "Failed to allocate a 536870928 byte allocation with 25165824 free bytes and 250MB until OOM, max allowed footprint 299427664, growth limit 536870912" (VmSize 5269896 kB)
2020-01-08 22:09:44.160 17361-17462/com.test.javacvapp W/System.err: Error on InputStream.reset() or skip(): java.lang.OutOfMemoryError: Failed to allocate a 536870928 byte allocation with 25165824 free bytes and 250MB until OOM, max allowed footprint 299427664, growth limit 536870912
2020-01-08 22:09:44.160 17361-17462/com.test.javacvapp I/mples.detectio: Starting a blocking GC Alloc
2020-01-08 22:09:44.160 17361-17462/com.test.javacvapp I/mples.detectio: Starting a blocking GC Alloc
2020-01-08 22:09:44.172 17361-17462/com.test.javacvapp I/mples.detectio: Alloc concurrent copying GC freed 14(31KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 261MB/285MB, paused 76us total 11.329ms
2020-01-08 22:09:44.172 17361-17462/com.test.javacvapp I/mples.detectio: Starting a blocking GC Alloc
2020-01-08 22:09:44.184 17361-17462/com.test.javacvapp I/mples.detectio: Alloc concurrent copying GC freed 1(15KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 261MB/285MB, paused 66us total 11.830ms
2020-01-08 22:09:44.184 17361-17462/com.test.javacvapp I/mples.detectio: Forcing collection of SoftReferences for 512MB allocation
2020-01-08 22:09:44.184 17361-17462/com.test.javacvapp I/mples.detectio: Starting a blocking GC Alloc
2020-01-08 22:09:44.382 17361-17462/com.test.javacvapp I/mples.detectio: Alloc concurrent copying GC freed 9(16KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 261MB/285MB, paused 64us total 198.092ms
2020-01-08 22:09:44.383 17361-17462/com.test.javacvapp W/mples.detectio: Throwing OutOfMemoryError "Failed to allocate a 536870924 byte allocation with 25165824 free bytes and 250MB until OOM, max allowed footprint 299427664, growth limit 536870912" (VmSize 5269896 kB)
...
--- REPEAT INDEFINITELY ---

This does not happen with a short 4K video (about 10s). I already tried to set the largeHeap property. I am on a Galaxy Note8 with Android 9.0 Pie. Videos don't get entirely loaded in RAM, aren't they?

--- EDIT

Source video is passed with the instructions

val inputVideo = contentResolver.openInputStream(mSelectedFileUri!!)
val grabber = FFmpegFrameGrabber(inputVideo)

If you don't need to seek within the InputStream, make sure to disable seeking.

I only need to analyze an mp4 video (800mb+) frame by frame. What do you mean by disabling seeking?

This is my code

val inputVideo = contentResolver.openInputStream(mSelectedFileUri!!)
val grabber = FFmpegFrameGrabber(inputVideo)
grabber.format = "mp4"
grabber.start()
var currentFrame = grabber.grabFrame(false, true, false, false)
while (currentFrame != null) {
  // perform frame processing
  Thread.sleep(300)

  mParsingFrameNumber += 1
  currentFrame = grabber.grab()
}

And hangs at grabber.start() (with OOM errors) when I load a "big" video file (seems 300mb+), how can this happen?

Sorry for my dumb questions but I really cannot figure this out. Thank you for your availability

As per issue #1304, set the maximumSize to 0 here:
http://bytedeco.org/javacv/apidocs/org/bytedeco/javacv/FFmpegFrameGrabber.html#FFmpegFrameGrabber-java.io.InputStream-

If I do so, grabber.grab() returns always null and in the log I have:

2020-01-09 16:03:37.652 24581-24600/com.test.javacv W/System: A resource failed to call close. 
2020-01-09 16:03:37.719 24581-24600/com.test.javacv E/System: Uncaught exception thrown by finalizer
2020-01-09 16:03:37.720 24581-24600/com.test.javacv E/System: org.bytedeco.javacv.FrameGrabber$Exception: Error on InputStream.close(): 
        at org.bytedeco.javacv.FFmpegFrameGrabber.releaseUnsafe(FFmpegFrameGrabber.java:245)
        at org.bytedeco.javacv.FFmpegFrameGrabber.release(FFmpegFrameGrabber.java:156)
        at org.bytedeco.javacv.FFmpegFrameGrabber.finalize(FFmpegFrameGrabber.java:265)
        at java.lang.Daemons$FinalizerDaemon.doFinalize(Daemons.java:250)
        at java.lang.Daemons$FinalizerDaemon.runInternal(Daemons.java:237)
        at java.lang.Daemons$Daemon.run(Daemons.java:103)
        at java.lang.Thread.run(Thread.java:764)
     Caused by: java.io.IOException: Resetting to invalid mark
        at java.io.BufferedInputStream.reset(BufferedInputStream.java:450)
        at org.bytedeco.javacv.FFmpegFrameGrabber.releaseUnsafe(FFmpegFrameGrabber.java:242)
        at org.bytedeco.javacv.FFmpegFrameGrabber.release(FFmpegFrameGrabber.java:156) 
        at org.bytedeco.javacv.FFmpegFrameGrabber.finalize(FFmpegFrameGrabber.java:265) 
        at java.lang.Daemons$FinalizerDaemon.doFinalize(Daemons.java:250) 
        at java.lang.Daemons$FinalizerDaemon.runInternal(Daemons.java:237) 
        at java.lang.Daemons$Daemon.run(Daemons.java:103) 
        at java.lang.Thread.run(Thread.java:764) 

I am using as dependency

implementation group: 'org.bytedeco', name: 'javacv-platform', version: '1.5.2'

So your format needs some seeking capabilities. You can try to set a small value for maximumSize. If that doesn't work, you will need to switch to a format that doesn't need to seek.

I think I've fixed all the memory leaks occurring in FFmpegFrameGrabber and FFmpegFrameRecorder.
Please give it a try with the snapshots: http://bytedeco.org/builds/

I think I've fixed all the memory leaks occurring in FFmpegFrameGrabber and FFmpegFrameRecorder.
Please give it a try with the snapshots: http://bytedeco.org/builds/

May I ask which version of this memory leak problem began to appear?

Was this page helpful?
0 / 5 - 0 ratings

Related issues

SenudaJayalath picture SenudaJayalath  ·  3Comments

RaGreen picture RaGreen  ·  4Comments

Bahramudin picture Bahramudin  ·  3Comments

newstarbka picture newstarbka  ·  5Comments

myScoopAndroidCamera picture myScoopAndroidCamera  ·  4Comments