Gatsby: Sharp causing segfault 'GLib-CRITICAL' in Netlify

[grantglidewell]

Description

related: https://github.com/gatsbyjs/gatsby/issues/6291
We had run into a similar issue, building on Netlify started to fail consistently after updating something related to gatsby-image.
our errors look like:

11:15:13 AM: (sharp:1397): GLib-CRITICAL **: 18:15:13.593: g_hash_table_lookup: assertion 'hash_table != NULL' failed
11:15:13 AM: (sharp:1397): GLib-CRITICAL **: 18:15:13.593: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed
11:15:13 AM: (sharp:1397): GLib-CRITICAL **: 18:15:13.593: g_hash_table_lookup: assertion 'hash_table != NULL' failed
11:15:21 AM: /usr/local/bin/build: line 34:  1397 Segmentation fault      (core dumped) gatsby build

we implemented the approaches in this comment adding

const sharp = require('sharp')

sharp.simd(false)
sharp.cache(false)

to our gatsby-node file. we also added a NODE_VERSION environment variable set to 12 in Netlify. Node 12 has an inherently higher memory cap. also added NODE_ENV as production.

Now in Netlify our builds don't immediately succeed, we have to clear the cache and rebuild.

Steps to reproduce

Our repo is private, however our dependencies in the project are:

{
    "@emotion/core": "^10.0.10",
    "@emotion/styled": "^10.0.10",
    "@svgr/webpack": "^4.2.0",
    "dotenv": "^7.0.0",
    "gatsby": "^2.8.6",
    "gatsby-image": "^2.2.4",
    "gatsby-plugin-emotion": "^4.0.6",
    "gatsby-plugin-google-tagmanager": "^2.0.15",
    "gatsby-plugin-manifest": "^2.0.28",
    "gatsby-plugin-offline": "^2.0.25",
    "gatsby-plugin-react-helmet": "^3.0.12",
    "gatsby-plugin-sharp": "^2.2.3",
    "gatsby-plugin-svgr": "^2.0.2",
    "gatsby-source-drupal": "^3.2.3",
    "gatsby-source-filesystem": "^2.0.29",
    "gatsby-transformer-sharp": "^2.1.18",
    "prop-types": "^15.7.2",
    "react": "^16.8.6",
    "react-dom": "^16.8.6",
    "react-helmet": "^5.2.0",
    "react-html-parser": "^2.0.2",
    "react-slick": "^0.24.0",
    "react-spring": "^8.0.25",
    "slick-carousel": "^1.8.1"
  }

Expected result

Builds should pass

Actual result

11:15:13 AM: (sharp:1397): GLib-CRITICAL **: 18:15:13.593: g_hash_table_lookup: assertion 'hash_table != NULL' failed 11:15:13 AM: (sharp:1397): GLib-CRITICAL **: 18:15:13.593: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed 11:15:13 AM: (sharp:1397): GLib-CRITICAL **: 18:15:13.593: g_hash_table_lookup: assertion 'hash_table != NULL' failed 11:15:21 AM: /usr/local/bin/build: line 34: 1397 Segmentation fault (core dumped) gatsby build

Environment

The environment is Netlify's xenial Image: https://github.com/netlify/build-image

[brohlson]

Same here - started happening to me last night. I think it could be likely be an issue with gatsby-transformer-sharp - which some of my failure messaging has pointed to. My builds will pass but only some percentages of the images will be transformed (around 20/150).

[grantglidewell]

I can force a pass if I clear cache and only have a single build going. but that defeats the purpose of ci/cd

[fool]

Hi folks, I work at Netlify and can shed some light on the matter:

That "success" is almost certainly not about cache clearing or anything except luck. The quoted segfault error is most likely your build running out of memory, which node handles fairly poorly at times; or maybe that's sharp's fault at the root of things, but regardless, I think it is a memory overuse issue.

So, why then would it behave differently at times?

Netlify's CI environment has a guarantee of only 3Gb of memory for your build - sometimes you might get up to 24Gb during a build, or during part of a build, if you get super lucky and don't have anyone else sharing your container host, but you need to aim for staying within 3gb on our lower level plans (free+Pro), since that is the RELIABLY available amount of memory. Higher cost plans can allow larger build environments, but we've coached dozens of customers through these issues with success and most have gotten their builds slimmed down and reliably working at Netlify. But I have more specific advice than "you must slim down!"

There are several plugins such as gatsby-image and gatsby-plugin-favicon, (those in particular are known agonists) that appear to use WAY more memory than that with some workloads cause build failures. While we haven't debugged in depth, it seems to potentially scale with the size or number of images you work with - we saw one person with a 5000 pixel wide favicon that was the only image, and it caused the build to blow up like this. He confirmed that the build used over 10Gb of RAM locally during that step. Again I am not certain this is the problem you're seeing, but that what seems to causing segfaults in general during builds using node (or, an explicit out of heap error from node), and those plugins (there may be others!) as far as we can tell the main problem, rather than it being a Netlify problem. You could try doing your build in 1CPU+3Gb mem under linux and see how it goes and I think you'd find the same thing...

There's supposedly some ways to offload image processing to other services that I've heard of working for some customers, or of course you could pre-process your images and check them in, instead of trying to do it during your Netlify build.

[grantglidewell]

This should probably be documented in the “build failure gotchas” in netlifys docs. I’ve been scouring for an explanation that fit these symptoms, so I’m happy to have that.

I know we’ve limited the cpu core count, I would assume there is also a way to add memory constraints to sharp (or its implementation in Gatsby-image)

[fool]

Good suggestion, thanks! I'll work on it with our docs team.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contributefor more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[cloderic]

We have this issue poping up randomly, is there a known workaround?

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[Temkit]

same issue here, any solution ?

(sharp:72549): GLib-GObject-WARNING **: 10:27:24.301: cannot register existing type 'VipsObject'

(sharp:72549): GLib-CRITICAL **: 10:27:24.303: g_once_init_leave: assertion 'result != 0' failed

(sharp:72549): GLib-GObject-CRITICAL **: 10:27:24.303: g_type_register_static: assertion 'parent_type > 0' failed

(sharp:72549): GLib-CRITICAL **: 10:27:24.303: g_once_init_leave: assertion 'result != 0' failed