Gatsby: Gatsby-image not processing some images (but those change on each build)

[benoitcalin]

Hi all,

It has been more than a week since this problem started when I tried to 'gatsby build'.

Some context first, I am working with WPGraphQL to get content and images from Wordpress, it was working fine but one day last week everything went wrong.

For a reason I cannot get, evey time I try to build my website, it fails because of some of my image having an 'imageFile' null.
Thing is, each build has a different image getting imageFile null, and one that was OK on the previous built can get null on this one.

The GraphQL queries are going through, I get the images url etc. so IMO it must be gatsby-image that might have upgraded in the last two weeks and my config is wrong now.

If you want to reproduce this error my repository is right here : https://github.com/benoitcalin/metroboulododo

I used this documentation to set my image processing : https://dev.to/nevernull/gatsby-with-wpgraphql-acf-and-gatbsy-image-72m
And I repeat it was working fine during two weeks.

If you want any complementary detail or info I would gladly give them, I am really stuck here and cannot send the website to production.

Thanks a lot in advance for your time.

Example of error on 'gastby build' (sorry couldn't find how to make it look better)

>{
  title: 'vidéo fromager',
  url: 'https://www.youtube.com/embed/XbB44SXtSVE'
}
{
  altText: 'portrait sylvain',
  sourceUrl: 'https://metroboulododo.fr/wp-content/uploads/2019/09/fromager-sylvain.png',
  imageFile: null

ERROR #95313

Building static HTML failed for path "/jobs/fromager/"

See our docs page for more info on this error: https://gatsby.dev/debug-html

  37 |     console.log(image)
  38 |     if (image) {
> 39 |       return image.imageFile.childImageSharp.fluid
     |                              ^
  40 |     } else { return { src: "https://metroboulododo.fr/wp-content/uploads/2019/09/ellipse-orange.png"} }
  41 |   }
  42 | 

WebpackError: TypeError: Cannot read property 'childImageSharp' of null

• job.js:39 Jobs.getImage
  src/templates/job.js:39:30

• job.js:161
  src/templates/job.js:161:35

• job.js:157 Jobs.render
  src/templates/job.js:157:30

[jasonbahl]

@benoitcalin thank you for reporting this issue. We’re looking into it.

What I believe might be happening is that the requests from
Gatsby to the WordPress server are happening so fast that the WordPress server is having trouble responding. My theory is that if we can throttle the requests and give the WordPress server some time to “breathe” between requests the issue might be resolved.

I’m looking into a few things and will follow up shortly.

[jasonbahl]

@benoitcalin I cloned your site and ran gatsby build 5 times with no issues. On the 6th time, I did get the following:

Errors:
  Unexpected error value: "failed to process https://metroboulododo.fr/wp-content/uploads/2019/09/anaya-1.jpg\nHTTPError: Response code 503 (Service Unavailable)"

  GraphQL request:134:15
  133 |               sourceUrl
  134 |               imageFile {
      |               ^
  135 |                 childImageSharp {
Plugin:
  none
Query:
  query GET_PAGE {
    wpgraphql {
      page(id: "cGFnZToxNjc=") {
        title
        page_home {
          buttonText
          banner {
            title
            subtitle
            image {
              altText
              sourceUrl
              imageFile {
                childImageSharp {
                  fluid(maxWidth: 1000) {
                    ...GatsbyImageSharpFluid
                  }
                }
              }
            }
          }
          about {
            title
            text2
            text1
            image1 {
              altText
              sourceUrl
              imageFile {
                childImageSharp {
                  fluid(maxWidth: 1000) {
                    ...GatsbyImageSharpFluid
                  }
                }
              }
            }
            image2 {
              altText
              sourceUrl
              imageFile {
                childImageSharp {
                  fluid(maxWidth: 1000) {
                    ...GatsbyImageSharpFluid
                  }
                }
              }
            }
          }
          features {
            title
            feature1 {
              title
              text
              image {
                altText
                sourceUrl
                imageFile {
                  childImageSharp {
                    fluid(maxWidth: 1000) {
                      ...GatsbyImageSharpFluid
                    }
                  }
                }
              }
            }
            feature2 {
              image {
                altText
                sourceUrl
                imageFile {
                  childImageSharp {
                    fluid(maxWidth: 1000) {
                      ...GatsbyImageSharpFluid
                    }
                  }
                }
              }
              text
              title
            }
            feature3 {
              image {
                altText
                sourceUrl
                imageFile {
                  childImageSharp {
                    fluid(maxWidth: 1000) {
                      ...GatsbyImageSharpFluid
                    }
                  }
                }
              }
              text
              title
            }
          }
          quote {
            image {
              altText
              sourceUrl
              imageFile {
                childImageSharp {
                  fluid(maxWidth: 1000) {
                    ...GatsbyImageSharpFluid
                  }
                }
              }
            }
            text
            title
          }
          testimonials {
            testimonial1 {
              image {
                altText
                sourceUrl
                imageFile {
                  childImageSharp {
                    fluid(maxWidth: 1000) {
                      ...GatsbyImageSharpFluid
                    }
                  }
                }
              }
              text
              title
            }
            testimonial2 {
              title
              text
              image {
                altText
                sourceUrl
                imageFile {
                  childImageSharp {
                    fluid(maxWidth: 1000) {
                      ...GatsbyImageSharpFluid
                    }
                  }
                }
              }
            }
            testimonial3 {
              title
              text
              image {
                altText
                sourceUrl
                imageFile {
                  childImageSharp {
                    fluid(maxWidth: 1000) {
                      ...GatsbyImageSharpFluid
                    }
                  }
                }
              }
            }
            title
          }
        }
      }
    }
  }

  fragment GatsbyImageSharpFluid on ImageSharpFluid {
    base64
    aspectRatio
    src
    srcSet
    sizes
  }

Based on the 503, it appears that the WordPress server is indeed struggling to respond, and I think the best way forward is to come up with a way to throttle the requests.

I need to look into the best way to throttle requests for the process that downloads the images. Will touch base again shortly.

[jasonbahl]

@benoitcalin You can throttle the concurrent download rate using the GATSBY_CONCURRENT_DOWNLOAD environment variable.

For example GATSBY_CONCURRENT_DOWNLOAD=1 gatsby build builds the site without issue, as it downloads one file at a time.

But GATSBY_CONCURRENT_DOWNLOAD=555 gatsby build causes issues, because the WordPress server is failing to respond to so many concurrent requests.

You can read more about using Environment variables with Gatsby here: https://www.gatsbyjs.org/docs/environment-variables/#example

@pieh has some thoughts on some internal changes we can make in the Gatsby codebase to make sure these issues are at least brought forward in a more clear manner so that if an external server is failing, such as 503 responses, we can surface that to users better.

[pieh]

Sometimes error happen during page query running and sometimes those 503 mentioned by @jasonbahl will happen in gatsby-node in https://github.com/benoitcalin/metroboulododo/blob/master/gatsby-node.js#L18-L19 which are silently ignored (this is how you end up with data that have null imageFile fields)

For now you could add something like

// GET JOBS INFOS
const results = await graphql(QUERY)

if (results.errors) {
  console.log(`errors running query`, results.errors)
  process.exit(1)
}

So those errors are not unhandled.

To workaround those 503 please use GATSBY_CONCURRENT_DOWNLOAD as mentioned in previous comment (probably you might want to adjust it to something more than 1 with trial and error so it's not crawlingly slow).

There few things that can be done on gatsby side of things:

• implement (smart) retry mechanism for our file downloader (if we get those 50x errors we don't want to immediately retry - we need some time off to let server catch up)
• don't hide those graphql errors in gatsby-node, we have this code https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/bootstrap/graphql-runner.js that should take care of it, but in this case it doesn't work as expected so this need to be fixed

[benoitcalin]

Thanks to both of you to take the time to look into it !
I thought it was WP server but could not find an appropriate way to handle this so your answers help me a lot :)

[wardpeet]

Thank you for opening this, @benoitcalin

We're marking this issue as answered and closing it for now but please feel free to reopen this and comment if you would like to continue this discussion. We hope we managed to help and thank you for using Gatsby! 💜