Sharp: Support and document how to configure sharp to fail when loading corrupt images

[mceachen]

Currently, sharp will only emit a warning for corrupt images. It'd be great to add an option to sharp to support rejecting the loading of these images.

See https://github.com/lovell/sharp/issues/607#issuecomment-298678935 and https://github.com/lovell/sharp/issues/607#issuecomment-298692269 for background.

(as always, thanks for Sharp! 👍

[jcupitt]

The property to set in vips is fail:

http://jcupitt.github.io/libvips/API/current/VipsForeignSave.html#VipsForeignLoad--fail

For example:

$ head -c 10000 k2.jpg > x.jpg
$ vips copy x.jpg y.jpg

(vips:30177): VIPS-WARNING **: read gave 2 warnings

(vips:30177): VIPS-WARNING **: VipsJpeg: Premature end of JPEG file
$ vips copy y.jpg z.jpg

So y.jpg is now a valid JPEG file, but with the bottom part set to a mid grey.

If you set fail, vips will bail out on the first warning:

$ vips copy x.jpg[fail] y.jpg

(vips:30579): VIPS-WARNING **: error in tile 0 x 64

(vips:30579): VIPS-WARNING **: error in tile 0 x 72
VipsJpeg: Premature end of JPEG file
VipsJpeg: out of order read at line 64
$ vips copy y.jpg z.jpg
VipsForeignLoad: "y.jpg" is not a known file format

So y.jpg is no longer a valid file.

From C++, fail is a gboolean property, just set it to TRUE like any load option.

[mceachen]

I'll take a crack at this now.

[mceachen]

I sketched out a PR, would love a review: https://github.com/lovell/sharp/pull/798

[mceachen]

OK, I'm missing something here. I picked up the PR a couple days ago, and can't get an invalid image to raise errors with VIPS, either from holes or from simple truncation, from either the command line tool, or by setting "fail".

$ for i in 1 2 4 8 16 32 ; do cp input.jpg punch-$i.jpg ; punch punch-$i.jpg $i ; done
$ for i in *punch* ; do vips copy $i[fail] /tmp/$i  ; done

No warnings are written by vips, and all files written to /tmp are valid JPGs of roughly the same size. This is with freshly-built vips 8.5.8 on linux.

I pushed what I expected to be a valid patch to https://github.com/lovell/sharp/pull/925. It's being built by https://travis-ci.org/mceachen/sharp/builds/269292587.

[jcupitt]

You're right, jpegload doesn't seem to handle fail properly, I'll have a look.

The png loader seems to have the obvious behaviour:

john@kiwi:~/pics$ head -c 10000 k2.png > x.png
john@kiwi:~/pics$ vips copy x.png y.png
(vips:17592): VIPS-WARNING **: Read Error
....
john@kiwi:~/pics$ echo $?
0

So lots of warnings, no error return, and y.png has some pixels in, though not many. With fail set:

john@kiwi:~/pics$ vips copy x.png[fail] y.png

(vips:18274): VIPS-WARNING **: Read Error

(vips:18274): VIPS-WARNING **: error in tile 0 x 0
vipspng: libpng read error
vips2png: unable to write "y.png"
john@kiwi:~/pics$ echo $?
1
john@kiwi:~/pics$ ls -l y.png
-rw-r--r-- 1 john john 54 Aug 28 19:22 y.png

Warnings, though not so many, an error return, and the output is not a valid PNG.

[jcupitt]

Ah, it's that libjpeg treats a truncated file as a warning, not an error, and libvips fail only applies to errors.

https://github.com/jcupitt/libvips/commit/942b0446a3fb81ef1d8eff7ffdf044514e0c761e#diff-10494e2c915fa105d119105af8a50976

I wonder if this is correct. A truncated file seems rather serious to me, and should perhaps be treated as an error.

[mceachen]

@jcupitt It seems like with that diff, you'll never see an error from libjpeg (and that's what my tests show).

Thanks for digging. I just added PNG tests to the PR, too.

[jcupitt]

With 8.5 head I now see:

john@kiwi:~/pics$ head -c 2000 k2.jpg > x.jpg
john@kiwi:~/pics$ vips copy x.jpg y.jpg
(vips:19972): VIPS-WARNING **: read gave 2 warnings
(vips:19972): VIPS-WARNING **: VipsJpeg: Premature end of JPEG file
john@kiwi:~/pics$ echo $?
0
john@kiwi:~/pics$ vips copy x.jpg[fail] y.jpg
(vips:20028): VIPS-WARNING **: error in tile 0 x 8
VipsJpeg: Premature end of JPEG file
john@kiwi:~/pics$ echo $?
1

I'll add a test to the new libvips test suite as well.

I think you could have got an error from libjpeg with --fail, but it would have been rather difficult.

[lovell]

sharp v0.19.0 now available, thanks for your work on this.