Gatsby: Number of generated thumbnails seems huge and crashes build in Windows

[kuworking]

Somehow a website of mine that had been building without problems, today has failed during build after having added a couple of additional posts

After trying to increase/decrease versions of sharp package and related plugins without success, I just upgraded all packages of my site just to see that both develop and build were failing

Then I just moved out all the site posts and develop again, and it worked again, but without content of course

My hypothesis is that the number of generated thumbnails gets huge at a point that it crashes due to a sort of out-of-memory error

 [====================----------] 13710/20646 439.4 s 66% Generating image thumbnailsException code=0xc0000005 flags=0x0 at 0x0000000000000000. Access violation - attempting to execute data at address 0x0000000000000000
error Command failed with exit code 3221225477.

Then, the question is how is this happening? In the sense that it wasn't happening 1 week ago without any significant change

Then,

• If I run my site with a bunch of static images and some non-static images, some pages and just 1 post, I have this

498 thumbnails, from ~60 images, I don't know if it is excessive but doesn't seem super-excessive

Then if I add a second post that refers to a static image (that was already there), then I have this

747 thumbnails, this must be wrong

Then if I add 3 more posts

1328 thumbnails

Then if I add the 38 other posts, that before compiled well in Netlify, then I have this

7636 thumbnails

which causes a failure when it went by ~2400

All of this is running gatsby clean && gatsby develop, so that cache and public folders are deleted

These are the packages.json (site -> gatsby-theme-B -> gatsby-theme-A)

  "dependencies": {
    "@mdx-js/mdx": "1.5.1",
    "babel-plugin-styled-components": "1.10.6",
    "gatsby-plugin-mdx": "1.0.24",
    "gatsby-plugin-sharp": "2.3.0",
    "gatsby-plugin-styled-components": "3.1.13",
    "gatsby-remark-copy-linked-files": "2.1.30",
    "gatsby-remark-images": "3.1.31",
    "gatsby-source-filesystem": "2.1.37",
    "gatsby-transformer-sharp": "2.3.5",
    "react-share": "3.0.1",
    "remark-slug": "5.1.2",
    "styled-components": "4.4.1"
  },
  "devDependencies": {
    "@mdx-js/react": "1.5.1",
    "gatsby": "2.17.15",
    "react": "16.12.0",
    "react-dom": "16.12.0"
  },

  "dependencies": {
    "@mdx-js/react": "^1.5.1",
    "deepmerge": "^4.2.2",
    "gatsby-image": "^2.2.33",
    "gatsby-plugin-feed": "^2.3.21",
    "gatsby-plugin-google-tagmanager": "^2.1.17",
    "gatsby-plugin-manifest": "2.2.28",
    "gatsby-plugin-offline": "3.0.21",
    "gatsby-plugin-page-creator": "2.1.30",
    "gatsby-plugin-react-helmet": "^3.1.15",
    "gatsby-plugin-robots-txt": "1.5.0",
    "gatsby-plugin-sitemap": "2.2.21",
    "gatsby-theme-A": "*",
    "react-cookie-consent": "2.5.0",
    "react-helmet": "^5.2.1",
    "styled-icons": "8.6.0"
  },
  "devDependencies": {
    "gatsby": "2.17.15",
    "react": "16.12.0",
    "react-dom": "16.12.0"
  },

  "dependencies": {
    "gatsby": "2.17.15",
    "react": "16.12.0",
    "react-dom": "16.12.0",
    "gatsby-theme-B": "*"
  },

[oduncan]

I had to downgrade to 2.15.15 because of this issue. I could have simply changed a word in a file and it would regenerate all images during bundle rebuild. Crashed Amazon EC2 instance with ENOMEM (memory error). Also it regenerated all images every time I ran develop, it doesn't do that in 2.15.15. It's as if it hashes improperly or fails some hash check every time it scans forcing it to regenerate all images regardless if they have changed or not.

[kuworking]

I had to downgrade to 2.15.15 because of this issue. I could have simply changed a word in a file and it would regenerate all images during bundle rebuild. Crashed Amazon EC2 instance with ENOMEM (memory error). Also it regenerated all images every time I ran develop, it doesn't do that in 2.15.15. It's as if it hashes improperly or fails some hash check every time it scans forcing it to regenerate all images regardless if they have changed or not.

In my case, I've just downgraded to 2.15.15 to give it a try and it crases equally again around ~2000 thumbnails

But before this wasn't happening, why all of the sudden it happens now??

[oduncan]

I should add that I didn't just downgrade the gatsby package, I rolled back my server to a snapshot I made before running npm update. Here are the associated package versions I rolled back to:

"gatsby": "^2.15.15",
"gatsby-image": "^2.2.19",
"gatsby-plugin-google-fonts": "^1.0.1",
"gatsby-plugin-manifest": "^2.2.16",
"gatsby-plugin-offline": "^2.2.10",
"gatsby-plugin-react-helmet": "^3.1.7",
"gatsby-plugin-sharp": "^2.2.22",
"gatsby-plugin-styled-components": "^3.1.9",
"gatsby-source-filesystem": "^2.1.22",
"gatsby-source-graphql": "^2.1.14",
"gatsby-transformer-sharp": "^2.2.14",

[kuworking]

I've just downgraded packages and it just works ...

Scattered in 3 package.json

    "gatsby-image": "2.2.19",
    "gatsby-plugin-feed": "^2.3.21",
    "gatsby-plugin-google-tagmanager": "^2.1.17",
    "gatsby-plugin-manifest": "2.2.16",
    "gatsby-plugin-offline": "3.0.21",
    "gatsby-plugin-page-creator": "2.1.30",
    "gatsby-plugin-react-helmet": "3.1.7",
    "gatsby-plugin-robots-txt": "1.5.0",
    "gatsby-plugin-sitemap": "2.2.21",

    "gatsby": "2.15.15",
    "react": "16.12.0",
    "react-dom": "16.12.0",

    "gatsby-plugin-mdx": "1.0.24",
    "gatsby-plugin-sharp": "2.2.22",
    "gatsby-plugin-styled-components": "3.1.13",
    "gatsby-remark-copy-linked-files": "2.1.30",
    "gatsby-remark-images": "3.1.31",
    "gatsby-source-filesystem": "2.1.22",
    "gatsby-transformer-sharp": "2.2.14",

These versions develop and build the site as before, also generating much lower amount of thumbnails

31 thumbnails only

Definitively makes much more sense! (thanks @oduncan !)

Is this a known behaviour? Do we have to change something in our gatsby-node to prevent this sort-of redundancy?

[kremalicious]

Can confirm, running into same issue on macOS where image generation just starts over and over during gatsby develop and saving a file. Also no image generation cache seems to be used anymore, images are always all generated fresh on every build.

It seems the culprit is gatsby-plugin-sharp v2.3.0 probably with this PR. As a workaround, downgrading gatsby-plugin-sharp to v2.2.39 fixed the issue for me.

[flikteoh]

We're also having the same issue generating more than ~17K+ images over and over since the latest update (macOS Catalina). Downgrading gatsby-plugin-sharp has temporary stopped the problem.

[wardpeet]

Published a new version [email protected] which reverts the code. It should be back to normal.

[kremalicious]

@wardpeet yup, working nicely as before, thanks for the quick response!

[ryanditjia]

It seems to me that 2.3.2 has the same problem 2.3.0 had.

2.3.2: 506 thumbnails
2.3.1: 80
2.3.0: 516
2.2.39: 84
2.2.x: 74

It brought up my build time from < 2 mins to > 6 mins, with only 6 or 7 images in the whole site.

[fodkastir]

yeah, I am having the same issue still.

[ryanditjia]

Forgot to say I’m experiencing the problem on my Mac and Netlify builds.

[wardpeet]

anyone able to share their repo? maybe in private? Feel free to give me an email at [email protected].

I'm thinking this is just a visual bug.

[ryanditjia]

https://github.com/angeloashmore/gatsby-source-prismic/issues/165#issue-526427561

In that issue you can see that my Netlify build time went up from 1:43 to 6:57.

The repo is private, I’ll invite you as collaborator and email you the .env.

[kremalicious]

I can not reproduce this bug anymore with [email protected] on 2 sites so all working nicely over here. Also, to be absolutely sure, installed dependencies from scratch by deleting node_modules and package-lock.json/yarn.lock.

[wardpeet]

thanks @ryanditjia for the repo. I've got a fix up on https://github.com/gatsbyjs/gatsby/pull/19717

To test it out please use npm install --save-dev gatsby-plugin-sharp@sharp-cache

[ryanditjia]

Thanks @wardpeet, it seems to fix the issue.

[muescha]

thanks @wardpeet also my error https://github.com/gatsbyjs/gatsby/issues/6291#issuecomment-556936215 at the gatsbyjs/gatsby/www is fixed with this

[wardpeet]

I'll publish a final fix on Monday

[wokejacqueline]

I'm on Mac 10.14.3, and my site has about 300 images, but gatsby-image-sharp is generating 55,000+/-. After upgrading from 2.1.3 to 2.3.2, the issue persists.

In running DEBUG=gatsby:*,graphql:* npm run build --verbose, it seems that the same image is being processed over and over again.

More unusually, the same block will repeat _again_ elsewhere in the file. I don't know anything about image processing, but it seems like it's making several thumbnails in a block, then doing it again...and again, and again, until my laptop is screaming. 😅

Thank you so much for looking into this. :)

[wardpeet]

Fixed in [email protected] sorry for the troubles

[agrawalaayushi]

@wardpeet, this version of gatsby-plugin-sharp is giving me this error-
(sharp:45876): GLib-GObject-WARNING **: 14:29:55.705: cannot register existing type 'VipsObject'

(sharp:45876): GLib-CRITICAL **: 14:29:55.705: g_once_init_leave: assertion 'result != 0' failed

(sharp:45876): GLib-GObject-CRITICAL **: 14:29:55.705: g_type_register_static: assertion 'parent_type > 0' failed

(sharp:45876): GLib-CRITICAL **: 14:29:55.705: g_once_init_leave: assertion 'result != 0' failed