Javacv: Automatic conversion to supported format in FFmpegFrameRecorder?

[b005t3r]

I have video frames in a specific format, which is not always supported by an encoder. Is there a way to instruct FFmpegFrameRecorder to convert the data from my (known) format to one of the desired formats before encoding? Or maybe a sub-automatic way? I tried going through supported formats for an encoder, like so:

avcodec.AVCodec enc = avcodec_find_encoder_by_name(encoder);
IntPointer formats = enc.pix_fmts();
// going through formats using get() and storing a format inside selectedFormat

and setting one of those in recorder, like so:

recorder.setPixelFormat(selectedFormat);

and recording frame like so:

recorder.record(frame, PIXEL_FORMAT); // pixel format is the format of frame 

It doesn't work, obviously. Am I doing it wrong?

[b005t3r]

And just to make sure, because it's not documented:

• for FrameGrabber: setPixelFormat() sets the output format, so I'll get Frames in that format)
• for FrameRecorder: setPixelFormat() sets the input format, so Frames have to be converted to that format before being encoded, but it's done internally, I just need to select a format that works with the selected codec

Am I right?

[saudet]

Yes, that's all correct and recorder.record(frame, PIXEL_FORMAT) does work. If you found a case when it doesn't work though, let's take a look at it.

[b005t3r]

Yeah, I guess it works just fine, was my mistake actually - I was using formats which won't work with sws_scale(). There's actually a function avcodec_find_best_pix_fmt_of_list() (terrific name, btw!) which selects a "best" format. I know use it and haven't experienced any crashes so far.

BUT, I think it's still not ideal and could maybe be incorporated into FFmpegFrameRecorder in the future (or as a separate entity). This automatic selection from ffmpeg often selects quite exotic formats, that I think might be problematic for some decoders (like mobiles). For instance, if my input format is BGR, it often selects YUV444P instead of YUV420P, which is much more common and no decoder will ever have problems with. So, yeah, works now, but isn't perfect.

[b005t3r]

One more thing I found about automatic format selection in ffmpeg - in short, yuv444p has been chosen as default for h264 and it's in general not handled by many widely used decoders :)

https://superuser.com/questions/874583/lossless-h-264-mp4-file-created-from-images-cannot-be-played-in-quicktime

This is probably not a terribly important issue and many have already found ways to work around it, but would be great to have some smarter auto format selection in the recorder, I think (something like recorder checks selected codec output formats, compares it with input format and selects whatever is best, which is probably yuv420p 99% of times :)

[saudet]

It's already defaulting to YUV420P: https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameRecorder.java#L555

[b005t3r]

Yeah, this doesn't work all the time, unfortunately (cost me yesterday to figure it out). Some codecs don't accept yuv420p (e.g. h264_qsv needs frames to be in nv12), so this simple one-liner won't work. You could change it to something more clever, like pick yuv420p if available, and if not, use whatever avcodec_find_best_pix_fmt_of_list() returns. I'm doing it in my test code, actually:

https://github.com/b005t3r/JavaCV-cli/blob/011f5f9c75e6fd762389af6e179447aa68806d71/src/main/java/com/lazarecki/Main.java#L107-L128

[saudet]

Sure, sounds great, please send a pull request!

[b005t3r]

We talked about it, man :) I'm not able to build it in a reasonable time nor do I know how to actually configure it to work with it. There's no step by step tutorial, I don't use Maven (and won't learn it anytime soon), so I really sorry, I won't be sending any pull requests. The best I can do to help fix bugs is to point them out and provide a code like that, sorry.

[saudet]

@mifritscher What about you? Would you be interested in working that?

[mifritscher]

@b005t3r: The lazy way is to have a "normal" Java project, which includes the JavaCV binary package, and copy the FFmpegFrameRecorder.java from the source package into the Java project. You can even rename it with minimal adaptions needed. To be honest, this was the way I developed most of the improvements ;-)

If I remember right I got even the tests running this way.

[saudet]

@b005t3r Could you open a pull request with that code? No need to build it or anything. I'll fix any build issues if they show up.

[b005t3r]

Sorry, i'm out for holidays, can't do it currently :(

wt., 16 cze 2020, 14:58 użytkownik Samuel Audet [e-mail]
napisał:

@b005t3r https://github.com/b005t3r Could you open a pull request with
that code? No need to build it or anything. I'll fix any build issues if
they show up.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bytedeco/javacv/issues/1183#issuecomment-644746050,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFY3HQEQDEPECZ6GACULBDRW5T7HANCNFSM4HE52I4Q
.

[saudet]

That's fine, no rush :) Thanks

[saudet]

@b005t3r What about now? :)

[b005t3r]

OK, so we're talking about this piece of code?

            avcodec.AVCodec enc = avcodec_find_encoder_by_name(encoder);

            IntPointer formats = enc.pix_fmts();

            int selectedFormat;

            int i = 0;
            while (true) {
                int format = formats.get(i++);

                // always prefer yuv420p, if available
                if(format == AV_PIX_FMT_YUV420P) {
                    selectedFormat = AV_PIX_FMT_YUV420P;
                    break;
                }
                if(format ==  AV_PIX_FMT_NONE) {
                    selectedFormat = avcodec.avcodec_find_best_pix_fmt_of_list(formats, PIXEL_FORMAT, 0, null);
                    break;
                }
            }

            recorder.setPixelFormat(selectedFormat);

And you want me to put it in where exactly? :) I guess I can add it to setVideoCodecName() as a automatic pixel format selection, if no pixel format is already set. Would that work for you?

[saudet]

Yes, and yes, sounds good! Thanks