Javacv: FFMpegFrameFilter crash when trying to pull audio

[k3rnL]

Hello,

I got a crash when I do a pullSamples(). If I do only pull(), I got a correct frame, but without audio. Is the normal behavior ?

Anyway, there is my code and the crash log :

        mSubtitleFilter = new FFmpegFrameFilter("drawtext=fontfile='C\\:/Windows/Fonts/Arial.ttf':text='Test and another test':x='(w-text_w)/2':y='(h-text_h)/2'", format.imageWidth, format.imageHeight);
        mSubtitleFilter.setPixelFormat(format.pixelFormat);
        mSubtitleFilter.setSampleFormat(format.videoSampleFormat);
        mSubtitleFilter.setAudioChannels(format.audioFormat.getChannels());
        try {
            mSubtitleFilter.start();
        } catch (FrameFilter.Exception e) {
            e.printStackTrace();
        }

        mFluxFormat = format;

        mExecutor = Executors.newSingleThreadScheduledExecutor();
        mExecutor.scheduleAtFixedRate(this::readFrameFromFilter, 5, 5, TimeUnit.MILLISECONDS);
    }

    private void readFrameFromFilter() {
        Frame frame = null;
        try {
//            frame = mSubtitleFilter.pull();
            frame = mSubtitleFilter.pullSamples();
        } catch (FrameFilter.Exception e) {
            e.printStackTrace();
        }

        if (frame != null) {
            feedOutputs(frame);
            if (frame.samples != null)
                System.out.println("coucou");
        }
    }

    @Override
    public void feed(Frame f) {
        try {
            mSubtitleFilter.push(f);
        } catch (FrameFilter.Exception e) {
            e.printStackTrace();
        }
    }

And the log :

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffb58dea9f0, pid=17548, tid=7860
#
# JRE version: Java(TM) SE Runtime Environment (10.0.2+13) (build 10.0.2+13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0.2+13, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# CAUDIO FORMAT ---- PCM_SIGNED 48000.0 Hz, 16 bit, stereo, 4 bytes/frame, big-endian
  0x00007ffb58dea9f0`

It's strange that the format announced by the crash is wrong. I have 6 channels. And 2 byte/frame.

[saudet]

To filter audio, you'll need to give it a second graph for that.

[k3rnL]

Ok, understood..

I can do without by buffering audio frame.

But I have another question, I need to change the filter after start(). Is it possible ? My attempts does not works for the moment.

[saudet]

Or if you don't need to filter audio, leave the number of audioChannels to 0.

[k3rnL]

Ok, thanks for your advice ! :)
I decided to push only frame containing images, I put others in a buffer that I read each time that I try to pull a frame. It works well.

Do you have an answer for my other question ?

But I have another question, I need to change the filter after start(). Is it possible ? My attempts does not works for the moment.

[saudet]

Just create another FFmpegFrameFilter, that's all!

[k3rnL]

Ok I will try later. But I'm little bit afraid about performance. I want to add subtitles from a file on a video flux. I need it to be responsive with the minimum of latency possible, because this flux is shown in a preview on the app and restreamed by internet. I can't have one second of delay.

[saudet]

If you need to change the text for each frame, I'd use something like Android Graphics, Java2D, or JavaFX.

[k3rnL]

Yes, after some try with FFmpegFrameFilter, it was not quick enough to ensure the stream fluidness. For example youtube refuse to show the stream.

Writing directly on the image buffer works much more better. For people who want to do it, there is how I get a Frame usable by FFmegFrameRecoder from a BufferedImage, from it's BufferedImage we can get a Graphics2D and draw on the image.
I use the Frame converter only one time to have better performance. But it's mandatory to use it because the buffer from FFmpegFrameGrabber is read only (at least, I wasn't able to use it.), the frame converter make a copy of this buffer in writable space.

`

        Java2DFrameConverter converter = new Java2DFrameConverter();
        BufferedImage img = converter.convert(frame); // This make a copy from the native c++ buffer

        Graphics2D g = img.createGraphics();
        g.fillRect(0,0,100,100);
        g.dispose();

        Frame frameToShow = f.clone(); // This don't copy the buffer. But copy image characteristics
        byte[] raw_buffer = ((DataBufferByte) img.getRaster().getDataBuffer()).getData();
        ByteBuffer buffer = ByteBuffer.wrap(raw_buffer);
        frameToShow.image[0] = buffer;

        feedOutputs(frameToShow); // Send the frame to stream and preview on the screen

`