Javacv: FFmpegFrameRecorder setVideoFilter - transpose or passthrough

[tomerhatav]

Is there a way to set transpose=dir=1:passthrough=portrait filters using FFmpegFrameRecorder.setVideoOption?
I'm trying to rotate the video recorded to portrait.
Saudet once said what can be done in command line tool "ffmpeg" can be done with FFmpegFrameRecorder, but I cant get the video to rotate.
I'm using RTMP link in the recorder URL.
Thank you,
Tomer

[saudet]

FFmpegFrameRecorder doesn't support avfilter, unfortunately, but this is something you could work on if you would like. A lot of people would be interested by this.

[skatash]

First of all Samuel many thanks for javacv code.
I can try to work on avfilter, but can you guide further which method or some example I can follow to start this task. At the moment I have read that Linesize can be useful to flip image in this class, but I am finding it hard to understand how can I achieve it ?

Can we implement any method in FFmpegFrameRecorder.recordImage() method to save rotated image in recorder ?

Does avfilter class meant for rotating video after recording or it can be used to rotate frame in FFmpegFrameRecorder.recordImage() ?

[saudet]

There's a couple of filtering samples here: http://ffmpeg.org/doxygen/trunk/examples.html
We would need to insert that into FFmpegFrameRecorder somewhere in the record() methods yes.

avfilter supports all these operations: https://ffmpeg.org/ffmpeg-filters.html
So once the code is in there, everything including rotate should work.

[skatash]

Thanks for your reply.

This is far above my paygrade and skill set !
If there would have been simpler solution like creating so file of some c files and using some native method to rotate image, you would have already given us.

All other cv methods like cvTranspose , cvCopy etc are giving very bad result.

In record() method we have AVFrame object and its being converted into BytePointer.
Isn't there any call we can make to manipulate AVFrame/BytePointer there and rotate/transpose it ?

Similary that method has sw_scale or av_fill like method ?
Isn't there any argument, property we can pass them to get rotated/transposed image ?

I already know the answer , it is not there, otherwise you would have recommended to every one already. But still I thought I should ask.

Actually dealing with C code is problem for me as I don't have any sense about that. Only Java and JNI.

[saudet]

@skatash The easiest way to use OpenCV is via OpenCVFrameConverter so be sure to use that. See issue #180 for an example of that.

[skatash]

To use OpenCVFrameConverter , in Android I will have to create yuvFrame using AndroidFrameConverter. Only this line to create frame take too much time, which results in loss of FPS in video and also if I apply other operation like crop or transpose it will take around 3 minutes to process a 10 second of video in standard mobile, even with using separate arraylist in new thread , which is way too much .

In Vine app, for VineRecorder those guys have used your Javacv code.
But they are very efficient timing wise and quality wise.
There is some magic surely they have done to crop and transpose each frame very efficiently.
They create 6 seconds of video in cropped and transposed in 6 seconds in a standard mobile.

[saudet]

I've added the FFmpegFrameFilter in the last commit. This currently works for video frames only. Sample usage below.

Initialization:

FFmpegFrameFilter filter = new FFmpegFrameFilter("transpose=cclock_flip", imageWidth, imageHeight);
filter.setPixelFormat(AV_PIX_FMT_NV21); // default camera format on Android
filter.start();

On each frame:

filter.push(yuvImage);
Frame frame;
while ((frame = filter.pull()) != null) {
    recorder.record(frame);
}

[skatash]

Thankyou a lot !
Does FFmpegFrameFilter is useful for cropping image at same time, for width and height both ?
Like ffmpeg command

crop=0:0:200:100 or crop=256:256:0:0

[saudet]

@skatash It should work. What error are you getting?

[aydinkn]

Hi @saudet ,

Can you compile JavaCV binary with FFMpegFrameFilter?

[saudet]

@aydinkn Just add the FFMpegFrameFilter.java to your own source files.

[aydinkn]

Thanks @saudet, It works perfectly :)
So, can we apply one or more filter at runtime?

[saudet]

@aydinkn Sure, are you getting some sort of error when you try to?

[aka-awasthi]

ffmpegframefilter not present in 1.0 javacv

[szitguy]

@saudet I init FFmpegFramwFilter like the following code:
mFilter = new FFmpegFrameFilter("crop=0:0:200:100", imageWidth, imageHeight);

But when i call FFmpegFrameFilter.start(), it failed.
Logcat shows FrameFilter$Exception: avfilter_graph_config() error.
Can you help me how to call crop command?

[aydinkn]

@szitguy I used FFmpegFrameFilter("transpose=clock") and it works well. Your issue looks like ffmpeg graph filter configuration.
Can you try named parameters like new FFmpegFrameFilter("crop=w=200:h=100:x=0:y=0") ?

[szitguy]

@aydinkn Thank you for your help! It works!
I really takes too much time on it.
So that's the way how to config FFmpegFrameFilter paramters.

[scheriff]

@saudet in order to use FFmpegFrameFilter should we compile javacv as told in the manual installation or is it possible to use with gradle?

[saudet]

@scheriff Like I said, just add the FFMpegFrameFilter.java to your own source files.

[tomerhatav]

Hello @saudet , i'm trying to use the filter but when i'm trying to .pull() a frame right after I did .push() I get this error "A/libc: Fatal signal 11 (SIGSEGV), code 1, fault addr 0x80 in tid 14055".
any idea why?
Thank you.

[saudet]

Does this happen only on Android? Or on another platform (Linux, Mac OS X, Windows) as well?

[tomerhatav]

@saudet I'm using only android with the latest javacv commits.
Nexus 5 marshmallow

[saudet]

So please try it on another platform, and let me know if the same exact code with the same exact data gives the same result or not, thank you!

[saudet]

Codecs don't support AV_PIX_FMT_NV21, but avfilter does, so just FrameFilter.setPixelFormat(AV_PIX_FMT_NV21), and call FrameRecorder.record(frame, AV_PIX_FMT_NV21) to indicate that the pixel format of the images is different from the codec.

[tomerhatav]

@saudet I successfully rotated the frames using the filter, I had to remove the setPixelFormat from the FFmpegFrameFilter,
But now I get a rotated grayscale frame. any idea why?

[saudet]

Ah, but I think I remember there is a bug in the pixel format conversion when the width and height of the images are the same as the width and height of the codec. If you try to specify a different width or height (say width + 1, height), and it doesn't crash, then this is a bug in FFmpeg. If this affects your application, you should report it upstream so they can fix it. Thank you

[saudet]

Because you didn't setPixelFormat()?

[tomerhatav]

Ok, problem fixed. thank you @saudet.

[saudet]

Well, it's not fixed if it crashes ;) If you report something upstream, let us know the link to the ticket here! Thanks

[Iqra786]

@saudet I init FFmpegFramwFilter like the following code:

new FFmpegFrameFilter("transpose=clock ,drawtext=text='text' ", imageWidth, imageHeight);

But when i call FFmpegFrameFilter.start(), it failed.

Logcat showsFrameFilter$Exception: avfilter_graph_parse_ptr().

Problem is with drawtext=text='text'. I tried with few option but no success.

Can you help me how to call add text command?

[saudet]

@Iqra786 Looks like it wants double quotes in there: https://ffmpeg.org/ffmpeg-filters.html#Examples-39

[tomerhatav]

@saudet when I use multiple filters like this : "transpose=clock ,crop=w=640:h=480:x=0:y=0"
I get this error "org.bytedeco.javacv.FrameFilter$Exception: avfilter_graph_config()"
Can you help me understand how to correctly use transpose and crop together?
Thank you.

[tomerhatav]

@saudet ok I swapped between the height and width and it works, maybe because i'm transposing it first. :)

[Iqra786]

@saudet I have tried with double quotes as following. Went through the documentation again but no success.
filter = new FFmpegFrameFilter("drawtext=\"text='hello world'\"", imageWidth, imageHeight);

[saudet]

@Iqra786 I see, we would need to build FFmpeg with libfreetype:

--enable-libfreetype     enable libfreetype, needed for drawtext filter [no]

Please open another issue about that.

[nikhilyadav]

I am trying to rotate Frame using FFMpegFilter as suggested by @saudet
following is my code but it is not working .

        filter = new FFmpegFrameFilter("transpose=cclock_flip", imageWidth+1, imageHeight);
        filter.setPixelFormat(avutil.AV_PIX_FMT_NV21); 



 if (yuvImage != null && recording) {
// Rotate the data for Portait Mode

                ((ByteBuffer) yuvImage.image[0].position(0)).put(data);

                    filter.push(yuvImage,avutil.AV_PIX_FMT_NV21);
                    while ((yuvImage = filter.pull()) != null) {
//                        recorder.record(yuvImage);
                        recorder.record(yuvImage,avutil.AV_PIX_FMT_NV21);
                    }

                } catch (FFmpegFrameRecorder.Exception e) {
                    Log.v(LOG_TAG, e.getMessage());
                    e.printStackTrace();
                } catch (FrameFilter.Exception e) {
                    e.printStackTrace();
                }
            }

Am I missing something .. Cause I still see my stream the same way. it is not showing me in portrait

[Iqra786]

@nikhilyadav - I can not see your filter.start() command.

[Iqra786]

@saudet
how can I apply following filter (https://ffmpeg.org/ffmpeg-filters.html#overlay-1)

logo, filter_complex , 'overlay=10:main_h-overlay_h-10

• logo is the png file but how can I get path for this logo.png from my asset file.
  I tried with following but get exception.
  "file:///android_asset/roboto/images.png -filter_complex 'overlay=10:main_h-overlay_h-10' "
  Thanks

[saudet]

@Iqra786 Assets are not directly accessible from native code. We need to extract them manually to some temporary directory like Context.getCacheDir().

[Iqra786]

Thanks @saudet
are following three filter are separated by comma.
"log , filter_complex , 'overlay=10:main_h-overlay_h-10' "

[nikhilyadav]

@Iqra786 I have started filter.start as soon as my FFMpegRecorder starts..

[Iqra786]

@nikhilyadav when are u init following line of code.
filter = new FFmpegFrameFilter("transpose=cclock_flip", imageWidth+1, imageHeight);
filter.setPixelFormat(avutil.AV_PIX_FMT_NV21);

[nikhilyadav]

@Iqra786

I first init filter and FFMpegrecorder..
Then I start both of these..
After that onPreviewFrame callback I do the push and pull from filter

[Iqra786]

which camera u are using for preview (back or front).
Might be it is working but u r not using right transpose.
try with one the following.
"transpose=cclock_flip", "transpose=clock" , "transpose=cclock" , "transpose=clock_flip".
Why are u adding + 1 in ImageWidth?

[nikhilyadav]

@Iqra786 It is mentioned above by @saudet .. to rectify a bug..
While I have tried both have .. It didn't work.
Let me try these and will get back to you

[saudet]

FFmpegFrameFilter is included in version 1.1. Thanks for reporting and your patience!

[CrazyOrr]

Hello @saudet
When I use crop filter like this: new FFmpegFrameFilter("crop=w=200:h=100:x=100:y=100"), it seems only w and h work, but x and y are not effective, which makes it not possible to crop an area that not start from the top-left corner. Is there a way to achieve this? Thank you.

[saudet]

@CrazyOrr That filtergraph works just fine here. It crops perfectly at the right x,y coordinates. :)

[CrazyOrr]

@saudet Yes, it works fine. It turns out that problem was when I use new FFmpegFrameFilter("crop=w:h:x:y", imageWidth, imageHeight), I mistakenly passed post-cropped witdh and height as imageWidth and imageHeight, which should be pre-cropped values. Thanks for your reply.

[1Dev11]

how can i cut video some frames like instagram can delete last one i using ffmpeg filter

[saudet]

@1Dev11 Just don't call record() for those frames.

[1Dev11]

how can i stop that i not getting idea

i have to make like instagram do , it can delete last recorded

i using ur method class

[saudet]

Could you provide an example?

[1Dev11]

yes wait i email u a my code

if u use instagram so u can see there video recording functionality there they provide delete functionality

[imalimin]

Why I can not use the FFmpegFrameFilter? @saudet
java.lang.UnsatisfiedLinkError: org.bytedeco.javacpp.postproc at java.lang.Class.classForName(Native Method) at java.lang.Class.forName(Class.java:308) at org.bytedeco.javacpp.Loader.load(Loader.java:413) at org.bytedeco.javacpp.Loader.load(Loader.java:381) at org.bytedeco.javacpp.avfilter.<clinit>(avfilter.java:17) at org.bytedeco.javacv.FFmpegFrameFilter.startUnsafe(FFmpegFrameFilter.java:169) at org.bytedeco.javacv.FFmpegFrameFilter.start(FFmpegFrameFilter.java:154)

[pachev]

Here's a working example if anyone is still searching for a solution. This would go in your onPreviewFrame method.

FFmpegFrameFilter filter = new FFmpegFrameFilter("transpose=1, transpose=1", imageWidth, imageHeight);
            filter.setPixelFormat(avutil.AV_PIX_FMT_NV21);
            try {
                filter.start();
            } catch (FrameFilter.Exception e) {
                e.printStackTrace();
            }

            /* get video data */
            if (yuvImage != null && recording) {
                ((ByteBuffer) yuvImage.image[0].position(0)).put(data);

                if (RECORD_LENGTH <= 0) try {
                    Log.v(LOG_TAG, "Writing Frame");
                    long t = 1000 * (System.currentTimeMillis() - startTime);
                    if (t > recorder.getTimestamp()) {
                        recorder.setTimestamp(t);
                    }
                     filter.push(yuvImage); //Runs the frame through the filter
                      Frame frame2;  
                    while ((frame2 = filter.pull()) != null) { //retrieve the filtered frame and send
                        recorder.record(frame2);
                    }
                } catch (FFmpegFrameRecorder.Exception e) {
                    Log.v(LOG_TAG, e.getMessage());
                    e.printStackTrace();
                } catch (FrameFilter.Exception e) {
                    e.printStackTrace();
                }
 }

[saudet]

@pachev Great! Could you make a sample out of this? Or simply update the RecordActivity sample and send a pull request? Thanks!

[saudet]

@lmylr Looks like I forgot to reply to this, but please follow the instructions in the README.md file and it will load.