Javacv: [android] Streaming video to rtmp ends with out of memory

[MarsVard]

I'm trying to stream a video to the facebook live api.

if I feed the recorder frames at 30 fps. (calling ffmpegrecorder.record() ), I shortly get an out of memory(512mb limit reached)

Is the recorder keeping all the frames in the memory? is there any way to clear these frames from memory after they have been sent to the server?

There is no prior warning or error, and the app is still responsive at the time the crash happens.

Caused by: java.lang.OutOfMemoryError: Failed to allocate memory within limits: totalBytes = 510M + 1M > maxBytes = 512M
                                                                                      at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:558)
                                                                                      at org.bytedeco.javacpp.Pointer.init(Pointer.java:121)
                                                                                      at org.bytedeco.javacpp.BytePointer.allocateArray(Native Method)
                                                                                      at org.bytedeco.javacpp.BytePointer.<init>(BytePointer.java:82)
                                                                                      at org.bytedeco.javacv.FFmpegFrameRecorder.recordImage(FFmpegFrameRecorder.java:778)
                                                                                      at org.bytedeco.javacv.FFmpegFrameRecorder.record(FFmpegFrameRecorder.java:756)
                                                                                      at org.bytedeco.javacv.FFmpegFrameRecorder.record(FFmpegFrameRecorder.java:749)

creation of recorder:

        recorder = new FFmpegFrameRecorder(pollConfig.getStreamUrl(), size, size, 2);
        recorder.setFormat("flv");
        recorder.setSampleRate(sampleAudioRateInHz);
        recorder.setInterleaved(true);
        recorder.setVideoOption("preset", "ultrafast");
        recorder.setVideoOption("crf", "28");
        recorder.setVideoBitrate(2000000);
        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
        recorder.setGopSize(GOP_LENGTH_IN_FRAMES);
        recorder.setFrameRate(frameRate);
        recorder.setAudioOption("crf", "0");
        recorder.setAudioQuality(0);
        recorder.setAudioBitrate(128000);
        recorder.setAudioChannels(2);
        recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);

        // create bogus audio to stream
        bufferSize = AudioRecord.getMinBufferSize(sampleAudioRateInHz, AudioFormat.CHANNEL_CONFIGURATION_STEREO, AudioFormat.ENCODING_PCM_16BIT);
        audioData = new short[bufferSize];
        for (int i = 0; i < audioData.length; i++) {
            audioData[i] = (short) i;
        }


        recorder.start();
        startTime = System.currentTimeMillis();

and every 33ms(30fps) I call record using this:

         frame = converterToBitmap.convert(bitmap);
         try {

//             Get the correct time
            Buffer[] buffer = { ShortBuffer.wrap(audioData, 0, audioData.length) };

            long t = 1000 * (System.currentTimeMillis() - startTime);
            if (t > recorder.getTimestamp()) {
                recorder.setTimestamp(t);
            }

            recorder.recordSamples(buffer);
            recorder.record(frame);


        } catch (FFmpegFrameRecorder.Exception e) {
            Log.v(LOG_TAG, e.getMessage());
            e.printStackTrace();
        } 

libraries used:

compile 'org.bytedeco:javacv:1.3'
    compile group: 'org.bytedeco.javacpp-presets', name: 'opencv', version: '3.1.0-1.3', classifier: 'android-arm'
    compile group: 'org.bytedeco.javacpp-presets', name: 'ffmpeg', version: '3.2.1-1.3', classifier: 'android-arm'

[saudet]

This was fixed with version 1.3. Please upgrade to version 1.3.

[MarsVard]

hi @saudet Sorry for the delay, I was just updating the question with library versions when you closed it :(

I'm using 1.3.

[saudet]

Ok, could you check the values you get for Pointer.physicalBytes() and Pointer.maxPhysicalBytes()?

[saudet]

I see, it's tripping on Pointer.maxBytes(). So, there are memory buffers that are still reachable. You'll need to find out where those are in your application. FFmpegFrameRecorder itself doesn't create new buffers for each frame...

[MarsVard]

My first thought was to check the bitmap creation method, so I tried using a single frame object and stream that frame over and over, but it still does the same thing.

[saudet]

Setting the "org.bytedeco.javacpp.logger.debug" system property to "true", and analyzing what you get in the log should help you understand where all those Pointer objects are getting allocated (but not deallocated).

[MarsVard]

Thanks! I'll try that now, I did log physicalBytes and maxPhysicalBytes and it's growing after record call, but like you said it might be allocated somewhere else.

System.setProperty("org.bytedeco.javacpp.logger.debug", "true");

@saudet I'm not getting any information in the logs :/ you mean the logcat right?

[MarsVard]

@saudet Could you please re-open this issue so somebody else can maybe pitch in?

[saudet]

Yes, set that property before calling anything related to JavaCPP. Messages will appear in Android's log.

[saudet]

BTW, if this used to work with an older version, set the "org.bytedeco.javacpp.maxphysicalbytes" system property to "0" during initialization to get back the old behavior.

[ninyess]

HI @saudet
I have same issue.

java.lang.OutOfMemoryError: Failed to allocate memory within limits: totalBytes = 510M + 1M > maxBytes = 512M
        at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:558)
    at org.bytedeco.javacpp.Pointer.init(Pointer.java:121)
    at org.bytedeco.javacpp.BytePointer.allocateArray(Native Method)
    at org.bytedeco.javacpp.BytePointer.<init>(BytePointer.java:82)
    at org.bytedeco.javacv.FFmpegFrameRecorder.recordImage(FFmpegFrameRecorder.java:778)
    at org.bytedeco.javacv.FFmpegFrameRecorder.record(FFmpegFrameRecorder.java:756)
    at org.bytedeco.javacv.FFmpegFrameRecorder.record(FFmpegFrameRecorder.java:749)

This problem occurs when repeatedly record a single frame rather than the camera's output.

mFrame = mConverter.convert(mBitmap); //not repeat

onPreviewFrame (
    mRecorder.record(mFrame); //loop
)

[saudet]

@ninyess Ok, so please try to also set the "org.bytedeco.javacpp.maxbytes" system property to "0".

[saudet]

You should also use JavaCPP 1.3.2 as this might be related to this bug: https://github.com/bytedeco/javacpp-presets/issues/380

[ninyess]

@saudet Sorry for the delay.
I guess I should try some more test, but I have solved this problem.
Use only one property, the error persists, but both settings are fixed.

System.setProperty("org.bytedeco.javacpp.maxphysicalbytes", "0"); System.setProperty("org.bytedeco.javacpp.maxbytes", "0");

Very Thanks!!!

[saudet]

Thanks for the feedback @ninyess!

[chrisbenincasa]

@saudet I'm running into a similar issue - I've followed the instructions here and set the system properties, however this just seems to let memory grow unbounded (could there possibly be a leak somewhere else?).

My code is quite simple:

val inputStream = new BufferedInputStream(new FileInputStream(file))
// Reduce logging from FFmpegFrameGrabber
av_log_set_level(AV_LOG_ERROR)
val grabber = new FFmpegFrameGrabber(inputStream)
val image = try {
  grabber.start()
  Option(grabber.grabImage()).flatMap(frame => Option(converter.convert(frame)))
} finally {
  grabber.stop()
  inputStream.close()
}

Any thoughts?

[saudet]

Sure, it's possible there's a memory leak. What does the profiler say about allocated Java objects, for starters?

[saudet]

BTW, I think I've fixed all the memory leaks occurring in FFmpegFrameGrabber and FFmpegFrameRecorder.
Please give it a try with the snapshots: http://bytedeco.org/builds/