Javacv: FFmpegFrameGrabber grab corrupted frame

[gkozyryatskyy]

Hello!

I have a video file and cutting image from it.
Here is a code example:

public static void testBlackLine() throws IOException {
        File outJpg = new File("out.jpg");
        long frameTime = 5000;
        try (FFmpegFrameGrabber video = new FFmpegFrameGrabber("https://www.dropbox.com/s/tu7m17pg9bvtkas/PorkBelly.mp4?dl=1")) {
            video.start();
            if (frameTime > 0) {
                video.setTimestamp(frameTime * 1000l); // microseconds
            }
            Frame frame = video.grabImage();
            BufferedImage image = new Java2DFrameConverter().convert(frame);
            ImageIO.write(image, VideoUtils.FRAME_IMAGE_EXTENSION, outJpg);
        }
    }

Everything seems like works fine, but out.jpg image has a black line at the right side... (https://www.dropbox.com/s/cvv2djhcl825xga/out.jpg?dl=0)
Input video looks fine (there is no this line).

• Now Im using javacv 1.3.3, but I tested it with javacv 1.3.4 builded from master brach and the line is still there.
• I tested it with different codec and nothing helps.
• I tried to cut this image with command line ffmpeg using command ffmpeg -i https://www.dropbox.com/s/tu7m17pg9bvtkas/PorkBelly.mp4?dl=1 -ss 00:00:5.000 -vframes 1 ffmpegout.jpg and everything fine there (there is no this black line)

Also this happening not with all videos, but just with "some of them". Do not know what exactly causing this black line.

So, maybe you have some ideas how I can get rid of this, when grabbing frame using FFmpegFrameGrabber?

Thank you!

[gkozyryatskyy]

Here is what I found..
If I adding video.setImageWidth(855); (+1 pixel from original width) before video.start(); picture is cut out with out "black line"...

[saudet]

Probably all codecs don't actually support odd width values like that
without some tricks. Could you check what the ffmpeg program does for that,
and we'll replicate it in FFmpegFrameRecorder?

[gkozyryatskyy]

@saudet
(This comment was updated because I was mistaken)

Original video size is 854. Size of the picture from ffmpeg program 854 also and it is with out black line.

If not using setImageWidth in FFmpegFrameGrabber (leave video width 854) the picture contains black line
If Im setting with - 1

video.start();
video.setImageWidth(video.getImageWidth() - 1);

image is fine and not containing black line... (with width - 2 picture also fine)

So seems like the problem not with odd values... And seems like changing width to any value, fix this problem somehow.

[saudet]

Does this happen with FFmpeg 3.4 as well?

[saudet]

The ffmpeg program uses SWS_BICUBIC by default, but FFmpegFrameGrabber uses SWS_BILINEAR:
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L771
SWS_BILINEAR might have a bug. You could try SWS_BICUBIC and see.

[gkozyryatskyy]

@saudet
FFmpeg 3.4 is part of javacv 1.3.4 and I try with 1.3.4 and the problem is still there..

What exactly option of FFmpegFrameGrabber should I use to set SWS_BICUBIC? I did not find anything like sws_flags..

[saudet]

You'll have to modify the source code...

[gkozyryatskyy]

@saudet
Yes, I found something here https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L769

[arianaa30]

How can I disable FFmpeg's default output?

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'video(1).mp4':
  Metadata:
    major_brand     : iso5
    minor_version   : 512
    compatible_brands: iso6mp41
    encoder         : Lavf58.0.102
  Duration: 00:00:04.60, start: 0.000000, bitrate: 2012 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 720x480 [SAR 8:9 DAR 4:3], Closed Captions, 1906 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      handler_name    : SoundHandler

[saudet]

@gkozyryatskyy Any updates on this? I've just released JavaCV 1.4.1, which uses FFmpeg 3.4.2. Maybe this is a bug that has been fixed upstream, so please try again with this new release!

@arianaa30 Just call av_set_log_level(AV_LOG_QUIET).

[gkozyryatskyy]

@saudet Just tested it with 1.4.1.. Black line is still there =(

Also for now I do not try to change pixel format...

[saudet]

@gkozyryatskyy Have you also tried SWS_BICUBIC?

[gkozyryatskyy]

@saudet not yet.. this is a minor bug from my side and it is reproduced not everywhere... So for now it is in backlog..

[saudet]

With JavaCV 1.4.3, we can now use SWS_BICUBIC easily by calling setImageScalingFlags(SWS_BICUBIC).
Let me know if that doesn't fix it though, thanks!

[gkozyryatskyy]

@saudet
Thank you a lot!
We already "fix" it from our side with rounding resolution, or something.. But we will try also to use SWS_BICUBIC in some time.

[svorlauf]

This problem still occurs in JavaCV 1.5. Changing the ImageScalingFlags does not help.
But it only appears when the frame width is not dividable by 8 leading to blocks with not all 8 pixels defined

[saudet]

@svorlauf You'll need to call setImageScalingFlags(SWS_BICUBIC) before start().

[svorlauf]

@saudet that is exactly what we do:
frameGrabber = new FFmpegFrameGrabber(file); frameGrabber.setImageScalingFlags(4); try { frameGrabber.start(); numFrames = frameGrabber.getLengthInFrames(); } catch (org.bytedeco.javacv.FrameGrabber.Exception e) { }
But it still uses wrong colors for the rightmost pixels

[saudet]

Ok, do you have a way to reproduce this?

[svorlauf]

example.zip
Yes we just create create the frameGrabber, call grabImage to get a frame and use the Java2DFrameConverter to convert it to BufferedImage and it is reproducable for all our videos with a width not dividable by 8

[saudet]

It happens only with Java2DFrameConverter? It doesn't happen with anything
else?

[svorlauf]

The Java2DFrameConverter is only for the next steps. We also printed out the pixel data of the picture_rgb in the frame. These already have 0 values at the end of every line. But i am not sure if it is also the case before calling sws_scale() in processImage() because that is where the image is converted from yuv to rgb

[saudet]

The stride can be larger than the width, that's not a problem. Make sure
you don't take pixels outside the width.

[svorlauf]

I rechecked the lineStride. It is 3 * the imageWidth. Exactly what to expect for rgb

[saudet]

It seems to be related to this open bug: https://trac.ffmpeg.org/ticket/1031

[svorlauf]

So initPictureRGB() should create a bigger buffer using the next bigger multiple of 8 for the width or lineStride?

[saudet]

Something like that, their workaround appears to work fine: https://github.com/bytedeco/javacv/commit/1d06eaa07186c2df59f2259bd8849421bbe6a40c

[svorlauf]

Works great. Thanks a lot. Is there any release planned in the near future? Or do i have to do run my own build to integrate the fix into the jar?

[saudet]

You could use the snapshots: http://bytedeco.org/builds/

[saudet]

JavaCV 1.5.1 has now been released with the fix! Enjoy