Javacv: Audio and video are out of sync

[egorapolonov]

Hi all,
Looks like the bug #870 appeared again.
The code example below extracts some part of video in time range.
If you compare original file and extracted fragment you can see that the fragment has out of sync problem: the sound comes earlier than some action happens on video.

import java.io.File;

import org.bytedeco.javacv.FFmpegFrameGrabber;
import org.bytedeco.javacv.FFmpegFrameRecorder;
import org.bytedeco.javacv.Frame;
import java.lang.Exception;

public class VideoExtractor {

    public static void main(String[] args) throws Exception {
        cutVideo(1631500000, 1661500000);
    }

    public static void cutVideo(long startTimeMicros, long endTimeMicros) throws Exception {
        File outputFile = new File("fragment_" + System.currentTimeMillis() + ".mp4");
        try (FFmpegFrameGrabber video = new FFmpegFrameGrabber("https://www.dropbox.com/s/255gqsm1wjn77ev/h264high_sound_issue.mp4?dl=1")) {
            video.start();
            video.setTimestamp(startTimeMicros, true); // doesn't matter in this example, produces same file
            try (FFmpegFrameRecorder rec = new FFmpegFrameRecorder(outputFile, video.getImageWidth(),
                    video.getImageHeight(), video.getAudioChannels())) {
                initializeEncoder(video, rec);
                rec.start();
                Frame frame;
                while (video.getTimestamp() <= endTimeMicros && (frame = video.grab()) != null) {
                    rec.record(frame);
                }
            }
        }
    }

    public static void initializeEncoder(FFmpegFrameGrabber grabber, FFmpegFrameRecorder recorder) {
        //video
        recorder.setVideoCodecName("libx264");
        recorder.setFrameRate(grabber.getFrameRate());
        recorder.setVideoBitrate(grabber.getVideoBitrate());
        recorder.setVideoOption("threads", "4");
        recorder.setVideoOption("profile", "main");
        recorder.setVideoOption("level","4");
        //audio
        recorder.setSampleRate(grabber.getSampleRate());
        recorder.setAudioBitrate(grabber.getAudioBitrate());
        recorder.setAudioCodec(grabber.getAudioCodec());
    }

}

[saudet]

@anotherche Would you have any ideas?

[anotherche]

I think that it is not a bug... I still think that the problem lies in the structure of audio and video streams in different files. You may read our conversation once again. There was a plot of audio-to-video lags along a test video clip. Timestamps of video and audio frames go not in exact sync between each other within a randomly cut piece of file. So, when you grab frames after setTimestamp you first obtain a frame (let it be a video frame) with that timestamp. Then next grabs may return you several following video frames and when an audio frame comes in that sequence it may correspond to a time lying up to 1 sec earlier than the previous video frames. To check this I would record a log during the cutting (inside the while loop) in which frame types and their timestamps should be saved for all the sequence of frames. If you will see that the first frame of different type (other than the very first frame in the whole sequence) comes with the timestamp less than the starting timestamp then this is the case I mean. In that case you may change the code so that to skip all the frames with timestampsendTime until a frame of different kind will come with timestamp>endTime. In other words, it seems that random cutting requires specific checking of the sync.

[saudet]

@anotherche He says the behavior changed, although it should be the same.

@egorapolonov What is the last version of JavaCV that this works correctly with?

[egorapolonov]

@saudet , @anotherche ,
This bug does not exist in 1.4.4, it appeared again in versions 1.5, 1.5.1 and 1.5.2

I tried to research code differences of FFmpegFrameGrabber and FFmpegFrameRecorder.
Possibly it was affected with changes in setTimestamp() method in the grabber, or sample_rate and time_base setup in the recorder.

[anotherche]

@egorapolonov, is the difference between sound and video growing with time, or is it constant? I guess if there were some problems with wrong sample_rate the difference would be growing.

[egorapolonov]

@anotherche ,

The difference looks like constant. I've uploaded fragment here https://www.dropbox.com/s/2qkc5ew43vvs5y6/fragment_1573823544018.mp4?dl=0

[saudet]

@egorapolonov The main difference that I can see between 1.4.4 and 1.5 is that multithreading is now enabled by default, which can result in the kind of behavior you are observing. If you need reproducibility, make sure to disable multithreading, for example, by calling:

        grabber.setVideoOption("threads", "1");
        grabber.setAudioOption("threads", "1");

[anotherche]

@egorapolonov, try this changed code of cutVideo function. Hope it will help to understand my thought about nature of the problem (and to solve the problem).

public static void cutVideo(long startTimeMicros, long endTimeMicros) throws Exception {
        File outputFile = new File("fragment_" + System.currentTimeMillis() + ".mp4");
        BufferedWriter outputWriter = null;
        outputWriter = new BufferedWriter(new FileWriter("log"));
        try (FFmpegFrameGrabber video = new FFmpegFrameGrabber("https://www.dropbox.com/s/255gqsm1wjn77ev/h264high_sound_issue.mp4?dl=1")) {
            video.start();
            video.setTimestamp(startTimeMicros, true); // doesn't matter in this example, produces same file
            try (FFmpegFrameRecorder rec = new FFmpegFrameRecorder(outputFile, video.getImageWidth(),
                    video.getImageHeight(), video.getAudioChannels())) {
                initializeEncoder(video, rec);
                rec.start();
                Frame frame;
              // 1.5 sec is added to ensure that both audio and video streams will be copied
              // this prevents skipping frames  with timestamp <= endTimeMicros which always appear after first 
              // appearance of a frame with timestamp = endTimeMicros
                while (video.getTimestamp() <= endTimeMicros + 1500000L && (frame = video.grab()) != null) { 

                    // this is just in order you could check that video and audio streams are lagging within ~0.3 sec in the source movie
                    // so when you simply cut the initial file from startTimeMicros to endTimeMicros you get audio and video not synced in final clip. That's normal!
                    outputWriter.write(frame.getTypes().toString() + " "+ Long.toString(frame.timestamp));
                outputWriter.newLine();

                    // this condition solves you problem forever :) It prevents copying frames with timestamp < startTimeMicros
                    // always appearing somewhat later and prevents copying frames with frame.timestamp > endTimeMicros
                    // occurring because of that additional 1.5 sec grabbing
                    if (frame.timestamp >= startTimeMicros && frame.timestamp <= endTimeMicros) 
                        rec.record(frame);
                }
                outputWriter.flush();  
            outputWriter.close(); 
            }
        }
    }

[anotherche]

You can also compare logs of recorded frames timestamp if move the outputWriter.write under the added condition.
I saved the two logs of frames recorded with or without the checking.
log-checked-recording.txt
log-simple-cut.txt

[egorapolonov]

Wow! Great! Thank you @anotherche , @saudet !

As quick as I check those solutions, I'll inform you.

[egorapolonov]

Hi @saudet , @anotherche

I'd like to thank you for your investigation and effort!

I've enabled single thread for recorder and grabber, both audio and video options

grabber.setVideoOption("threads", "1");
grabber.setAudioOption("threads", "1");
recorder.setVideoOption("threads", "1");
recorder.setAudioOption("threads", "1");

This didn't help, so the multithreading feature isn't a root cause.

Then I used changed code and logs provided by @anotherche .
Definitely, that's the right solution! Thank you for such detailed explanation!

[anotherche]

Fine! This should be a kind of instruction on “how to cut clips from a video correctly”

[saudet]

It's possible that new versions of FFmpeg decode streams a bit differently, so yes if it works fine by checking the timestamp of all frames, then it's still working within specs. :)

[saudet]

@anotherche That would make a great wiki page indeed. Please feel free to create one here:
https://github.com/bytedeco/javacv/wiki

[anotherche]

OK. I will think on this when I will feel that I'm free to :)