Javacv: Questions about FFmpegFrameGrabber and FFmpegFrameRecorder using InputStream and OutputStream

[lonelyzerg]

I use a ByteArrayOutputStream bos to get encoded video from FFmpegFrameRecorder recorder, and send it though UDP. in another thread I receive the packets and writ the data into a PipedOutputStream pos. There is a PipedInputStream pis connected it with pos which is used as input for the FFmpegFrameGrabber grabber.

When i run the code, there is firstly a 214 byte header transmitted through UDP. After the recorder encoded 58 frames, the UDP socket start sending the first frame. But it was until 128 frames written to pos(that is 186th frame encoded by the recorder) when the grabberis opened.

This process takes about 13 seconds, and then the grabber starts showing the 128 frames at 1000 fps. after it catch up with the recorder (still 58 frames earlier) it displays at a normal speed.

I tried to debug and find out that ret = avformat_open_input(oc, filename, f, options) in FFmpegFrameGrabber.java:573 is the reason for taking so much time to start the grabber.

So my questions are:

1. How to send the frames in real time?
2. How to shorten the time needed to excute avformat_open_input(oc, filename, f, options);

Here is part of my code:

//the code to record and send video stream through udp:
DatagramSocket ds = new DatagramSocket();
DatagramPacket dp;
byte[] send;
int count = 0;
System.out.println("recorder started");
while (frame.isVisible()) {
Frame data = grabber.grab();
frame.showImage(grabber.grab());
if (startTime == 0) 
    startTime = System.currentTimeMillis();
videoTS = 1000 * (System.currentTimeMillis() - startTime);
recorder.setTimestamp(videoTS);
recorder.record(data);
System.out.println("recording " + count);
if (bos.size() > 0) {
    send = bos.toByteArray();
    dp = new DatagramPacket(send, send.length, loc, 10010);
    ds.send(dp);
    System.out.println("Data sent through udp: " + send.length);
    bos.reset();
}

//the code to receive data from UDP and write to pos:
byte[] buffer = new byte[100000];
DatagramSocket ds = new DatagramSocket(10010);
DatagramPacket dp = new DatagramPacket(buffer, 100000);
int count = 0;
while (flag) {
    ds.receive(dp);
    pos.write(dp.getData(), 0, dp.getLength());
    pos.flush();
    System.out.println(count);
    count++;
    //System.out.println("Data sent to showimg..." + dp.getLength());
}

//the code to read frame from pis and show it:
FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(pis);
grabber.setFormat("flv");
grabber.setVideoCodec(avcodec.AV_CODEC_ID_H264);
grabber.start();

[saudet]

That's perfectly normal. If you need to display at a constant rate, use a timer.

[saudet]

The delay is determined by the GOP size, so reduce that to reduce the delay.

[lonelyzerg]

@saudet Thanks for your answer. I tried some different values and it do helps. I wonder if there is a function that can disable B-frames like -bf 0 in ffmpeg? And even if I lowered the gop size, it takes the same time for avformat_open_input() to execute. Is that normal?

[saudet]

There are options for B-frames and input buffers somewhere, yes. Please
refer to FFmpeg's documentation. We should be able to set them with
setVideoOption() and setOption(). And if you'd like to make a contribution
based on your findings, it would be most welcome!

[saudet]

FYI, here are the names of all relevant options:
https://ffmpeg.org/ffmpeg-codecs.html
https://ffmpeg.org/ffmpeg-formats.html
There might also be more undocumented ones...

[lonelyzerg]

Thanks! I have solved the latency problem by setting “tune” as “zerolatency”

[saudet]

Great! Please make a contribution by documenting this somewhere.

[skeshavan]

I have little control over the encoding part (which is not done by ffmpeg). Is there any way to reduce latency from the decoding end (in ffmpegframegrabber)? I think this is possible because I am able to open the first h264 file (contains data for one compressed frame) in vlc which contains the iframe.

[skeshavan]

I tried things like the following but nothing worked:

                grabber.setVideoOption("analyzeduration", "1");
                grabber.setVideoOption("fflags", "nobuffer");
                grabber.setVideoOption("probesize", "32");
                grabber.setVideoOption("sync", "ext");

EDIT: Trying the setOption API worked...

[srajca]

@skeshavan Can you please elaborate a bit for me?