Javacv: Unable to create an audio+video file with H264 codec

[avanibhatnagar]

I am able to create a video file with H264 codec, but when I add audio to it using this method and then try setting the codec to H264, it just creates an empty file, and keeps on running.

The vidPath below has a H264 codec video. The audioPath is an aac audio file.
This is my method to add audio to a video:

    FrameGrabber grabber1 = new FFmpegFrameGrabber(vidPath);
    FrameGrabber grabber2 = new FFmpegFrameGrabber(audioPath);
    grabber1.start();
    grabber2.start();
    FrameRecorder recorder = new FFmpegFrameRecorder(outputPath,
            grabber1.getImageWidth(), grabber1.getImageHeight(),
            grabber2.getAudioChannels());
    recorder.setFormat("mp4");
    recorder.setVideoQuality(10);
    recorder.setFrameRate(grabber1.getFrameRate());
    recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);

    recorder.start();
    while ((frame1 = grabber1.grabFrame()) != null ||
            ((frame2 = grabber2.grabFrame()) != null)) {
        recorder.record(frame1);
        recorder.record(frame2);
    }
    recorder.stop();
    grabber1.stop();
    grabber2.stop();
}

This above method works without adding this: recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
and creates a MPEG4 video with audio added to it perfectly. Runs infinitely and creates empty file when codec set to H264 only.

Is there any way I can write a method that converts a MPEG4 codec file (video+audio) to H264 codec or VP8 or VORBIS?

[saudet]

This works fine here with JavaCV 1.4.2 on Linux, what is your platform?

Could you also try to call setVideoOption("threads", "1")?

[avanibhatnagar]

When I call setVideoOption("threads","1"), I get this:
at line 101, i.e. recorder.record(frame1);

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:67)
Caused by: org.bytedeco.javacv.FrameRecorder$Exception: avcodec_encode_video2() error -542398533: Could not encode video packet.
at org.bytedeco.javacv.FFmpegFrameRecorder.recordImage(FFmpegFrameRecorder.java:957)
at org.bytedeco.javacv.FFmpegFrameRecorder.record(FFmpegFrameRecorder.java:866)
at org.bytedeco.javacv.FFmpegFrameRecorder.record(FFmpegFrameRecorder.java:859)
at com.creatics.web.IAmVideoController.addMP3toMovie(IAmVideoController.java:101)
at com.creatics.web.IAmVideoController.main(IAmVideoController.java:54)
... 5 more
[libx264 @ 0000000019e0f040] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0000000019e0f040] profile High 4:4:4 Predictive, level 3.1, 4:2:0 8-bit
[libx264 @ 0000000019e0f040] 64 - core 152 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=0 mixed_ref=1 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=1 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0
Output #0, mp4, to 'C:\Users\Avani Bhatnagar\Documents\ffmpeg trial\ffmpeg java class\VideoRate.mp4':
Metadata:
encoder : Lavf58.12.100
Stream #0:0: Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 640x720, q=2-31, 9 kb/s, 17408 tbn
[libx264 @ 0000000019e0f040] frame I:4 Avg QP: 0.00 size:182324
[libx264 @ 0000000019e0f040] mb I I16..4: 52.4% 0.0% 47.6%
[libx264 @ 0000000019e0f040] 8x8 transform intra:0.0%
[libx264 @ 0000000019e0f040] coded y,uvDC,uvAC intra: 95.2% 94.1% 94.0%
[libx264 @ 0000000019e0f040] i16 v,h,dc,p: 53% 46% 1% 0%
[libx264 @ 0000000019e0f040] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 40% 39% 2% 3% 4% 3% 4% 3% 2%
[libx264 @ 0000000019e0f040] i8c dc,h,v,p: 6% 50% 44% 0%
[libx264 @ 0000000019e0f040] kb/s:495.92
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C:\Users\Avani Bhatnagar\Documents\ffmpeg trial\ffmpeg java class\output trial\A\VideoRate.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
encoder : Lavf56.37.100
Duration: 00:00:11.77, start: 0.000000, bitrate: 183 kb/s
Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 640x720 [SAR 1:1 DAR 8:9], 182 kb/s, 0.34 fps, 0.34 tbr, 17408 tbn, 17 tbc (default)
Metadata:
handler_name : VideoHandler
[aac @ 00000000217d5fc0] Estimating duration from bitrate, this may be inaccurate
Input #0, aac, from 'C:\Users\Avani Bhatnagar\Documents\ffmpeg trial\ffmpeg java class\Downtown_20SecondVersion_Final.aac':
Duration: 00:00:19.93, bitrate: 128 kb/s
Stream #0:0: Audio: aac (LC), 44100 Hz, stereo, fltp, 128 kb/s
[libx264 @ 0000000021736740] -qscale is ignored, -crf is recommended.
[libx264 @ 0000000021736740] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0000000021736740] profile High, level 3.1
[libx264 @ 0000000021736740] 264 - core 152 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=1 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=1 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=10.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'C:\Users\Avani Bhatnagar\Documents\ffmpeg trial\ffmpeg java class\output trial\A\Movie\Movie.mp4':
Metadata:
encoder : Lavf58.12.100
Stream #0:0: Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 640x720, q=2-31, 400 kb/s, 17408 tbn
Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 64 kb/s
[libx264 @ 0000000021736740] lookahead thread is already stopped

Process finished with exit code 1

[avanibhatnagar]

I'm using JavaCV 1.4.2 on Windows
Right when I change the codec back to MPEG4, it works.

These are the only dependencies I need, correct?

    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacv</artifactId>
        <version>1.4.2</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacv-platform</artifactId>
        <version>1.4.2</version>
    </dependency>

[saudet]

That's caused by passing multiple null frames to record(). Use && instead of || in your loop:

        while ((frame1 = grabber1.grabFrame()) != null &&
                ((frame2 = grabber2.grabFrame()) != null)) {

[avanibhatnagar]

Ok so changing it to && creates a video but the audio played for less than a second right when I play it and then it stops... with MPEG4 and H264

Changing it back to || works for MPEG but not H264

[saudet]

You'll need to record all the audio frames to get all of the audio :) They are not equal in time to video frames.

[avanibhatnagar]

I see. Thanks a lot! I'm new to FFMPEG. Would setting frame rate help? Is there a sample code I can look at?

[avanibhatnagar]

The exact same code works perfectly fine with MPEG4 codec. Only converting it to another codec is causing the problem..

[avanibhatnagar]

Is there a way I can just convert a MPEG codec movie(audio(aac)+video) to a H264?
Something like this? I know it is incorrect, but can you guide me how to make it work? What would I have to add/remove from this?

    File file = new File(filePath); //this file would have the final Codec H264 movie
    FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(outputPath); //this would be the MPEG 4 codec movie that needs to be converted
    frameGrabber.start();
    FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(file,640,720);
    recorder.setFormat("mp4");
    recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
    recorder.setFrameRate(frameGrabber.getFrameRate());
    //recorder.setSampleFormat(frameGrabber.getSampleFormat());
    recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
    recorder.setVideoQuality(10); 
    recorder.start();
    recorder.stop();

[avanibhatnagar]

Any ideas @saudet ?

[saudet]

The idea with JavaCV is to process frames, so a loop like this should work:

    while ((frame1 = grabber1.grabFrame()) != null ||
            ((frame2 = grabber2.grabFrame()) != null)) {
        if (frame1 != null) recorder.record(frame1);
        if (frame2 != null) recorder.record(frame2);
    }

But if you're only interested in converting video files from one format to the other, I would recommend using the ffmpeg command line tool.

[avanibhatnagar]

I am doing this over the server, so I will be unable to use FFMPEG command line. I used JAVACV, to convert images to a video using FFMPEG frame recorder and then add audio to it. The video is H264 codec but when I pass it to my method to add audio to it as shown in my method above, it does so perfectly and creates a .mp4 with MPEG 4 codec, but when I use the same to set to H264 codec so that it can be displayed on all browsers, I get avcodec_encode_video2() error -542398533: Could not encode video packet.

[avanibhatnagar]

I think it is just like issue #745 https://github.com/bytedeco/javacv/issues/745

[avanibhatnagar]

I was able to pass a file with the MPEG4 codec and convert it to H264 finally!!
The only problem now is a few frames in the end get cut, any suggestions?

[saudet]

Calling recorder.record(null) might help, but that's already done on
stop()...

[saudet]

In any case, please provide some data to reproduce this, and I'll look into it! Thanks

[Orcunkazan]

I was able to pass a file with the MPEG4 codec and convert it to H264 finally!!
The only problem now is a few frames in the end get cut, any suggestions?

Did you really do that? Is it possible that a video would use the H264 codec for the audio stream?