Sharp: AVIF support in prebuilt binaries

[asilvas]

AVIF is coming soon to the most popular browsers. Is there a reason this is still considered experimental and not ready for production? https://sharp.pixelplumbing.com/api-output#heif

[lovell]

Hi Aaron, I expect to be able to include aom+libheif in the prebuilt binaries in order to support AVIF input and output, but the following things will need to exist first, in approximately this order:

• [x] 1. Addition of aom as a dependency in the libheif fuzzer - PR merged - https://github.com/google/oss-fuzz/pull/4414
• [x] 2. Addition of aom+libheif as dependencies in the libvips fuzzer - PR merged - https://github.com/google/oss-fuzz/pull/4463
• [x] 3. Addition of aom+libheif in the prebuilt libvips binaries provided by sharp
  
  
  
  • Linux/macOS https://github.com/lovell/sharp-libvips/issues/64 dependent on https://github.com/strukturag/libheif/pull/354
  
  
  • Windows https://github.com/libvips/build-win64-mxe/issues/13 dependent on https://github.com/strukturag/libheif/pull/337
  
  
• [x] 4. Addition of unit tests in sharp for aom+libheif+libvips combo to cover various dimensions and colour spaces etc.

Anything you can do to help with the first two security-related items will be much appreciated.

(The patent-encumbered HEIC codec will not be supported in the prebuilt binaries.)

[asilvas]

Thanks! I'll close.

[lovell]

I'm going to reopen this to become the "canonical" issue to track adding AVIF support in the prebuilt binaries.

For anyone else coming to this issue, sharp already supports AVIF, you need to compile libvips from source with support for libheif that has been compiled with support for aom/rav1e, before installing sharp.

[lovell]

I've created https://github.com/google/oss-fuzz/pull/4414 to enable fuzz testing of AVIF images with libheif, see step 1 above. Once approved and merged, my plan is to let that run for a couple of weeks to give it a chance to find any problems and get them fixed before moving onto step 2.

[soukicz]

my plan is to let that run for a couple of weeks to give it a chance to find any problems and get them fixed before moving onto step 2.

Is that something that would benefit from having some monster AWS EC2 server? I don't know much C but I do have credit card :)

[lovell]

@soukicz OSS-Fuzz runs on Google's ClusterFuzz hardware, currently >25k cores of processing power.

[lovell]

The addition of AVIF fuzzing to libvips is now merged at https://github.com/google/oss-fuzz/pull/4463 - let's see what it finds in the next few days before moving onto step 3.

[lovell]

There's lots of interest in this issue so I'd like to remind everyone that if you work at a for-profit organisation, and you are not already doing so, then please donate to https://opencollective.com/libvips if you benefit commercially from sharp and libvips.

[lovell]

Prebuilt binaries containing libvips, libheif and aom have now been successfully built for Linux (x64, ARMv6, ARMv7, ARM64), macOS (x64) and Windows (x64, x86). A groot big thank you to @kleisauke for help with these.

This means those feeling brave may want to carry out some early/alpha testing.

npm install lovell/sharp#avif

Given installation is from GitHub rather than npm, the above command will require node-gyp and its dependencies (C++ compiler, Python) - see https://sharp.pixelplumbing.com/install#building-from-source

The following code samples are made possible when using these pre-release binaries:

// AVAILABLE FROM v0.27.0+

await sharp(input).toFile("out.avif");

await sharp("in.avif").toFile("out.jpg");

await sharp(input).avif({ quality: 30, speed: 1 }).toBuffer();

// AVAILABLE FROM v0.27.0+

Temporary docs for the forthcoming avif() can be viewed at https://github.com/lovell/sharp/blob/avif/docs/api-output.md#avif

Please do not use these in production systems yet.

[lovell]

Fuzz tests added for this issue are (un)fortunately still finding security bugs in AVIF-related dependencies, approximately one every ~5 days, so this this not yet ready for release. Please do not use these in production systems yet.

[virgofx]

@lovell Any chance you could shed some more insight into your last comment? If there are a few random bugs would it be possible to still utilize the avif format in a release and then have subsequent build releases address security concerns in libvips? I guess at some level there will always be issues with dependencies. If the sharp app works well with the format yet there are a few downstream issues, couldn't those just be updated and re-released in minor patch builds?

[lovell]

@virgofx If your question can be rephrased as "Will pre-compiled binaries be provided that contain known security vulnerabilities?" then the answer is "no".

[ascorbic]

@lovell Is there an area where time/resources could be invested to get this over the line, or is this just lots of bugs in various dependencies?

[tomgallagher]

As a side issue, I had read that encoding and decoding AVIF files was very slow. Are there any performance stats on this with libvips?

[lovell]

@ascorbic There are a number of security-related fixes in libheif and libvips awaiting release.

There are also a couple of open bugs relating to the use of clap (clean aperture) boxes:

• https://github.com/strukturag/libheif/issues/365 (e.g. images smaller than 16x16px won't work)
• https://github.com/libvips/libvips/issues/1808

[AndreiIgna]

When this is shipped it will make sharp even more helpful than already is 🎉

[lovell]

libheif v1.10.0 and libvips v8.10.5 have now been released, plus (sadly but understandably) all the pubs in London are pandemically-shut, which means we're looking at this side of Christmas for a release of sharp that includes prebuilt binaries that support AVIF. :tada:

[lovell]

If anyone would like to test the v0.27.0-beta1 pre-release, which includes prebuilt binaries for all the usual platforms, please use the following (this doesn't require a compiler, but does involve a git checkout of the whole repo so will still be slower than an npm-published package):

$ npm install lovell/sharp

I plan to publish this to npm as v0.27.0 tomorrow.

[lovell]

sharp v0.27.0 is now available with prebuilt binaries that contain libaom and libheif for full AVIF decoding and encoding support.

Please note the current lack of support in most browsers for AVIF images smaller than 16x16.

https://sharp.pixelplumbing.com/api-output#avif

If anyone discovers a problem image, please open a new issue and provide as much information as possible, ideally trying libheif's command line tools such as heif-convert first to ensure it is not an upstream issue - https://github.com/strukturag/libheif#example-programs

[joehoyle]

Amazing, thanks so much @lovell, I had done some work to update https://github.com/humanmade/tachyon/pull/130 to the sharp branch, sorry I didn't manage to get beta-1 testing in time. So far everything looks to be working well!

[lovell]

Here's a basic CLI tool to quickly/easily transcode a directory of images into AVIF, which might be useful to help assess the suitability of AVIF for a particular task.

https://www.npmjs.com/package/avif

Run npx avif --help for usage instructions.

[lovell]

Reminder: if you work at a for-profit organisation that benefits commercially from sharp and libvips then please tell your manager to become a backer at https://opencollective.com/libvips

[ascorbic]

Reminder: if you work at a for-profit organisation that benefits commercially from sharp and libvips then please tell your manager to become a backer at https://opencollective.com/libvips

That's awesome news @lovell. I'll certainly be raising this.

[ascorbic]

As a note to anybody using Apple Silicon who wants AVIF support: bear in mind that before compiling libvips from source you will need to compile and install libheif and aom from source too. Using homebrew doesn't work either, and it will either fail to include AVIF support, or it will include it but fail to generate images. You must first compile and install aom from git, then compile and install libheif (checking that it has found aom when running configure), then finally compile and install libvips.