Gatsby: vips2png: "unable to write to target" error when generating thumbnails breaks the build

[dcorb]

Description

During the build step, this error happens

vipspng: libpng read error
vips2png: unable to write to target

Please, could you provide me with a way to debug which image might be breaking vips2png (used by sharp plugin)? We have a thousand of them

The error happens in CircleCI, and it's difficult to replicate in our local because of x number of reasons.

More context:

info bootstrap finished - 39.546s

warning 

 [] 0/96 0.0 s 0% Generating image thumbnailswarning 
                 The requested height "500px" for a resolutions field for
                 the file /home/circleci/project/.cache/caches/gatsby-source-drupal/7f6345e68e32e87cf256a6dbc8a1b279/example.png
                 was larger than the actual image height of 330px!
                 If possible, replace the current image with a larger one.
 [] 0/135 0.0 s 0% Generating image thumbnails [] 0/182 0.0 s 0% Generating image thumbnails [] 0/185 0.0 s 0% Generating image thumbnails [] 1739/1749 34.5 s 99% Downloading remote files [] 1741/1749 34.6 s 99% Downloading remote fileserror There was an error in your GraphQL query:

vipspng: libpng read error 
vips2png: unable to write to target

  57 |   langcode
  58 |   title
  59 |   created
  60 |   path {
  61 |     alias
  62 |   }
  63 |   relationships {
  64 |     field_recipe_image {
  65 |       localFile {
  66 |         childImageSharp { 
> 67 |           fixed(height: 500)  
     |           ^
  68 |             ...GatsbyImageSharpFixed_withWebp
  69 |           }
  70 |         }
  71 |       }
  72 |     }
  73 |   }
  74 | }
  75 | 
  76 | query ListingQuery {
  77 |   allNodeProductPage {

File path: /home/circleci/project/src/components/Listing.js
Plugin: none

not finished run queries - 3.854s
not finished Generating image thumbnails - 3.828s

Steps to reproduce

Sorry, not easy to reproduce, since I'm trying to pinpoint what PNG image is the one that is breaking sharp plugin (specifically, what is breaking vips2png )
It seems it's an open issue https://github.com/libvips/libvips/issues/1578

Expected result

Build succeeds.

Actual result

Build breaks.

Environment

System:
    OS: macOS Mojave 10.14.5
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 12.12.0 - /usr/local/bin/node
    Yarn: 1.19.1 - /usr/local/bin/yarn
    npm: 6.11.3 - /usr/local/bin/npm
  Languages:
    Python: 2.7.17 - /usr/local/bin/python
  Browsers:
    Chrome: 83.0.4103.61
    Firefox: 75.0
    Safari: 12.1.1
  npmPackages:
    gatsby: ^2.22.3 => 2.22.3
    gatsby-background-image: ^1.1.1 => 1.1.1
    gatsby-image: ^2.4.5 => 2.4.5
    gatsby-plugin-client-side-redirect: ^1.1.0 => 1.1.0
    gatsby-plugin-manifest: ^2.4.8 => 2.4.8
    gatsby-plugin-meta-redirect: ^1.1.1 => 1.1.1
    gatsby-plugin-offline: ^3.2.6 => 3.2.6
    gatsby-plugin-react-helmet: ^3.3.2 => 3.3.2
    gatsby-plugin-sharp: ^2.6.8 => 2.6.8
    gatsby-plugin-styled-components: ^3.3.2 => 3.3.2
    gatsby-source-drupal: ^3.5.7 => 3.5.7
    gatsby-source-filesystem: ^2.3.7 => 2.3.7
    gatsby-transformer-json: ^2.4.3 => 2.4.3
    gatsby-transformer-sharp: ^2.5.3 => 2.5.3
  npmGlobalPackages:
    gatsby-cli: 2.8.18

[dcorb]

Actually, more thoughts: reading the https://github.com/libvips/libvips/issues/1578 issue, the description says that happens randomly (like it happen to us too), and it could be that is related to concurrency. (another hypothesis opens here)
I would like to try concurrency 1, but the value can't be modified externally. See this line from gatsby sharp plugin
https://github.com/gatsbyjs/gatsby/blob/4fd158ffc1b613a3af30a922acba627552077e9b/packages/gatsby-plugin-sharp/src/process-file.js#L23

[ascorbic]

@dcorb I don't think it's the same as that issue. Your first error says vipspng: libpng read error, which means it is almost certainly a broken source image as you first thought. The problem here is that we should fail gracefully, and should at the very least log out which image is causing the error.

[dcorb]

Thanks @ascorbic. It's happening randomly, some of the builds are green. We are thinking it could be some of the images are huge, 40Mb could error silently. We will try to narrow down the issue.

[dcorb]

related : https://github.com/gatsbyjs/gatsby/issues/11996 "Improve reliability of gatsby-plugin-sharp by moving Sharp image transformations into child process"

[dcorb]

very very related https://github.com/gatsbyjs/gatsby/issues/13442 (We have the same issue, some times there is not even errors, but the build fails). We have a thousand images, some of them huge.

[dcorb]

The root issue is confirmed: too big files, and Gatsby (or Sharp) not able to give proper Error messages, some times no error messages at all.

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[fernandezllambi]

I'm having the same issue and it's not related to the image size (in my case). I happens randomly on different images (all png) and sizes from 176KB to 6MB. Which is weird because it fails on just png files but in my case, consistently on the same files. Maybe there is somthing with my files, I'll dig into that but closing the issue due to inactivity is not the solution

Having more detail about the error would be great

[rvanmarkus]

I'm seeing the same errors here; it happens randomly as well (both on Mac OS and CI).
Sometimes the error is:
vips2png: unable to write to target
the other time it's:
Input file has corrupt header:

In both cases there is no image path in the errors; so it's really hard to track down if a certain image input from the source is the problem. At least having that would maybe already help us find out the cause of the problem.

[jcupitt]

I'm sorry this is causing trouble :( libvips 8.10 will try to display more useful error messages for png load and save, see:

https://github.com/libvips/libvips/commit/70c0b36d4f34d690a8060369d96871a8338f8db7
https://github.com/libvips/libvips/commit/17cddc196b9c6e24fba7dd9594da9f9ecade08cb

Perhaps it'll help find the cause. The linked issue seems to be related to Alpine's tiny default stack, but that doesn't seem to be the issue here.

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[jlarmstrongiv]

I’m running into this issue with an svg file as well (on MacOS), specifically with the gatsby-plugin-manifest icon.

[jcupitt]

8.10 is out now and should have better error messages. For example, I see:

✔ ~/try 
04:42 $ vips copy IMG_2601.jpg x.png
✔ ~/try 
04:42 $ vips copy x.png x.png
pngload: end of stream
vips2png: unable to write to target x.png
✘-1 ~/try 
04:42 $ vips --version
vips-8.10.0-Sat Jul 18 13:45:47 UTC 2020

Is there anything useful in the error log?

(source and dest must be different files, so this triggers an error)

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[TristanPeers]

Fully agree that there needs to be some more info to show which images are causing the error. This is all we get from our build process:

$ node --stack-size=4096 $(command -v gatsby) build success open and validate gatsby-configs - 0.055s success load plugins - 1.168s success onPreInit - 0.163s success delete html and css files from previous builds - 0.003s success initialize cache - 0.006s success copy gatsby files - 0.045s success onPreBootstrap - 0.016s success createSchemaCustomization - 0.005s error UNHANDLED REJECTION pngload: non-recoverable state vips2png: unable to write to target target Error: pngload: non-recoverable state vips2png: unable to write to target target not finished source and transform nodes - 23.643s not finished Generating image thumbnails - 2.905s error Command failed with exit code 1.

Had to build an Ubuntu ec2 instance (VIPS wouldn't install via Brew on Mac), upload all the PNG's from our site and run vips copy against each one to find the problematic ones. Far from ideal.

[KyleAMathews]

Could you share one of the problematic images? I'd love to get a better error message.

[TristanPeers]

Sadly these images contained sensitive company information so I cannot upload them.

The png's basically had corrupt headers, they looked and opened fine in Mac preview, but vips didn't like them:

$ vips copy internet-egress-privatelink.png /tmp/
(vips:15721): VIPS-WARNING **: 11:42:42.765: zTXt: incorrect header check

I can look on Monday to see if I can recreate/find a suitable image to upload. Cheers

[jcupitt]

The sPNG developers would be interested too -- they are always on the lookout for exotic test files.

Would Preview let you draw on the image to mask out any sensitive info?

[TristanPeers]

I believe so, yes. I tried re-exporting the png's with Preview and the error was still present so this should allow me to blank out the images as you suggest. I resorted to exporting them as JPEG in the end.

[TristanPeers]

Hi - I think I was mistaken when saying re-exporting them preserved the error… I cannot recreate that now unfortunately.

Anyway I managed to find a broken png ( (vips:36610): VIPS-WARNING **: 10:30:23.602: zTXt: incorrect header check ) which doesn't have any sensitive info:

[jcupitt]

Thanks for finding this dodgy PNG. It works for me with libpng:

john@yingna ~/pics $ vips copy broken.png x.jpg
(vips:2389214): VIPS-WARNING **: 10:33:48.084: zTXt: incorrect header check

Though, as you say, there are some warnings.

With libvips configured with libspng 0.6.1 as the reader, you get an error:

john@yingna ~/pics $ vipsheader broken.png 
broken.png: 341x415 uchar, 4 bands, srgb, pngload
john@yingna ~/pics $ vips copy broken.png x.jpg
pngload: non-recoverable state

Could that be what you're seeing?

I tried with git master libspng and it also fails. I'll open an issue on the libspng issue tracker.

[KyleAMathews]

Traced down the root source of the error & catch the error and print out which file is causing the problem https://github.com/gatsbyjs/gatsby/pull/28614

[TristanPeers]

Fantastic - many thanks @KyleAMathews