gatsby-source-shopify unable to complete build

[OllieJT]

Description

I have been unable to complete the build process using the 'gatsby-source-shopify' plugin. The progress varies each time, but it always stalls on "source and transform nodes" after a successful "fetched and processed collections"

Steps to reproduce

I have tried

• disabling all plugins just to get through a build
• using a fresh install with gatsby-starter
• using the shopify-starter (a forked version of gatsby-shopify) which works with the demo content, but not with my own store credentials.

I am beginning to believe it is something to do with the store I am pulling from. The store is a live site with ~550 products and so I am apprehensions about posting any keys.

Expected result

• gatsby develop
• gatsby-source-shopify sources and transforms nodes
• build completes and is in development mode

Actual result

• gatsby develop
• gatsby-source-shopify begins to source and transforms nodes but freezes

vbi-store $  gatsby develop

success open and validate gatsby-configs — 0.025 s
success load plugins — 0.212 s
success onPreInit — 1.370 s
success initialize cache — 0.199 s
success copy gatsby files — 0.062 s
success onPreBootstrap — 0.895 s
gatsby-source-shopify/visuals-by-impulse starting to fetch data from Shopify
 source and transform nodes
gatsby-source-shopify/visuals-by-impulse fetched and processed productTypes: 551.321ms
gatsby-source-shopify/visuals-by-impulse fetched and processed policies: 570.847ms
Downloading remote files [==============================] 21/21 1.0 secs 100%
gatsby-source-shopify/visuals-by-impulse fetched and processed collections: 2024.666ms
source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/stream-panels-colored-twitch-panels-1.png
source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/stream-panels-colored-twitch-panels-6.png
source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/Logo_Presentation_62426fbf-eed8-4710-92b8-443e8cca67d3.png
source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/16x9-Green_b9f770a0-1409-4b2b-885c-cd31c37a155d.png
source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/16x9-Teal_5c4ae163-68be-4670-8106-69eeefffdedf.png

source and transform nodes

Failed to process remote content varies from 0-10 items

Environment

System:
OS: macOS 10.14.3
CPU: (8) x64 Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 10.15.3 - /usr/local/bin/node
Yarn: 1.10.1 - /usr/local/bin/yarn
npm: 6.9.0 - /usr/local/bin/npm
Languages:
Python: 2.7.10 - /usr/bin/python
Browsers:
Chrome: 73.0.3683.86
Safari: 12.0.3
npmPackages:
gatsby: ^2.3.4 => 2.3.4
gatsby-image: ^2.0.35 => 2.0.35
gatsby-plugin-manifest: ^2.0.26 => 2.0.26
gatsby-plugin-offline: ^2.0.25 => 2.0.25
gatsby-plugin-prefetch-google-fonts: ^1.4.1 => 1.4.1
gatsby-plugin-prismic-preview: ^2.0.0 => 2.0.0
gatsby-plugin-react-helmet: ^3.0.11 => 3.0.11
gatsby-plugin-sharp: ^2.0.32 => 2.0.32
gatsby-plugin-styled-components: ^3.0.7 => 3.0.7
gatsby-remark-prettier: ^1.0.0 => 1.0.0
gatsby-source-custom-api: ^0.1.1 => 0.1.1
gatsby-source-filesystem: ^2.0.28 => 2.0.28
gatsby-source-graphql: ^2.0.15 => 2.0.15
gatsby-source-prismic-graphql: ^3.0.2 => 3.0.2
gatsby-source-prismic-rich-text-fields: ^1.0.0 => 1.0.0
gatsby-source-shopify-min: github:olliejt/gatsby-source-shopify-min#master => 2.0.24
gatsby-transformer-remark: ^2.3.8 => 2.3.8
gatsby-transformer-sharp: ^2.1.17 => 2.1.17
npmGlobalPackages:
gatsby-cli: 2.4.14

[OllieJT]

Additional tests.

• deployed shopify-starter and built successfully with demo data
• changed gatsby-source-shopify2 to gatsby-source-shopify and reinstalled all node_modules
• built successfully with demo data
  
  
  
  • cleared gatsby cache and changed .env credentials to the desired store
  
  
• ERROR (see below)

success open and validate gatsby-configs — 0.008 s
success load plugins — 0.210 s
success onPreInit — 0.586 s
success initialize cache — 0.005 s
success copy gatsby files — 0.070 s
success onPreBootstrap — 0.005 s
⠁ 
gatsby-source-shopify/undefined starting to fetch data from Shopify
⢀ source and transform nodes
error an error occured while sourcing data
query: 
  """

      query GetCollections($first: Int!, $after: String) {
        shop {
          collections(first: $first, after: $after) {
            pageInfo {
              hasNextPage
            }
            edges {
              cursor
              node {
                description
                descriptionHtml
                handle
                id
                image {
                  altText
                  id
                  src
                }
                products(first: 250) {
                  edges {
                    node {
                      id
                    }
                  }
                }
                title
                updatedAt
              }
            }
          }
        }
      }

  """
variables: 
  first: 250
  after: null
success source and transform nodes — 0.844 s
success building schema — 0.169 s
error gatsby-node.js returned an error


  TypeError: Cannot read property 'allShopifyProduct' of undefined

  - gatsby-node.js:16 graphql.then.result
    /Users/olliejt/Documents/GitHub/gatsby-shopify-starter/gatsby-node.js:16:17


success createPages — 0.058 s
success createPagesStatefully — 0.065 s
success onPreExtractQueries — 0.006 s
success update schema — 0.028 s
error GraphQL Error Encountered 2 error(s):
- Unknown field 'allShopifyProduct' on type 'Query'. Source: document `usersOlliejtDocumentsGitHubGatsbyShopifyStarterSrcComponentsProductGridIndexJs3449166288` file: `GraphQL request`

  GraphQL request (3:9)
  2:       query {
  3:         allShopifyProduct(
             ^
  4:           sort: {

- Unknown field 'shopifyProduct' on type 'Query'. 

      file: /Users/olliejt/Documents/GitHub/gatsby-shopify-starter/src/templates/ProductPage/index.js



success extract queries from components — 0.139 s
⠁ (node:88029) DeprecationWarning: Passing lineNumber and colNumber is deprecated to @babel/code-frame. Please use `codeFrameColumns`.
error 
The GraphQL query from /Users/olliejt/Documents/GitHub/gatsby-shopify-starter/src/components/ProductGrid/index.js failed.

Errors:
  Cannot query field "allShopifyProduct" on type "Query".

  GraphQL request (2:3)
  1: query usersOlliejtDocumentsGitHubGatsbyShopifyStarterSrcComponentsProductGridIndexJs3449166288 {
  2:   allShopifyProduct(sort: {fields: [createdAt], order: DESC}) {
       ^
  3:     edges {

Plugin:
  none
Query:
  query usersOlliejtDocumentsGitHubGatsbyShopifyStarterSrcComponentsProductGridIndexJs3449166288 {
    allShopifyProduct(sort: {fields: [createdAt], order: DESC}) {
      edges {
        node {
          id
          title
          handle
          createdAt
          images {
            id
            originalSrc
            localFile {
              childImageSharp {
                fluid(maxWidth: 910) {
                  ...GatsbyImageSharpFluid_withWebp_tracedSVG
                }
              }
            }
          }
          variants {
            price
          }
        }
      }
    }
  }

  fragment GatsbyImageSharpFluid_withWebp_tracedSVG on ImageSharpFluid {
    tracedSVG
    aspectRatio
    src
    srcSet
    srcWebp
    srcSetWebp
    sizes
  }

success run graphql queries — 0.089 s — 9/9 101.58 queries/second
success write out page data — 0.008 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.119 s

info bootstrap finished - 9.278 s

 ERROR  Failed to compile with 1 errors                                                                                                                            2:37:10 AM

 error  in ./src/components/ProductGrid/index.js

Module Error (from ./node_modules/eslint-loader/index.js):

/Users/olliejt/Documents/GitHub/gatsby-shopify-starter/src/components/ProductGrid/index.js
  11:9  error  Cannot query field "allShopifyProduct" on type "Query"  graphql/template-strings

✖ 1 problem (1 error, 0 warnings)


 @ ./src/pages/index.js 9:0-52 15:163-174
 @ ./.cache/sync-requires.js
 @ ./.cache/app.js
 @ multi event-source-polyfill (webpack)-hot-middleware/client.js?path=/__webpack_hmr&reload=true&overlay=false ./.cache/app

✖ ｢wdm｣: 
ERROR in ./src/components/ProductGrid/index.js
Module Error (from ./node_modules/eslint-loader/index.js):

/Users/olliejt/Documents/GitHub/gatsby-shopify-starter/src/components/ProductGrid/index.js
  11:9  error  Cannot query field "allShopifyProduct" on type "Query"  graphql/template-strings

✖ 1 problem (1 error, 0 warnings)

 @ ./src/pages/index.js 9:0-52 15:163-174
 @ ./.cache/sync-requires.js
 @ ./.cache/app.js
 @ multi event-source-polyfill (webpack)-hot-middleware/client.js?path=/__webpack_hmr&reload=true&overlay=false ./.cache/app
ℹ ｢wdm｣: Failed to compile.

[OllieJT]

Another update, I am using the gatsby-source-graphql to connect to the same store with no issues so far. I'm still not sure what this could be caused by.

[violy]

Observing on my project a same pattern warning Failed to process remote content, due to requests overload, try to change concurrent: 200 to lower value, like 10 in create-remote-file-node.js of gatsby-source-filesystem plugin.

https://github.com/gatsbyjs/gatsby/blob/d74b3f08c747cec150ebd62638e0c4f3f8c2aa8f/packages/gatsby-source-filesystem/src/create-remote-file-node.js#L89-L103

I hope it will help you.

[OllieJT]

Thankyou @violy there's definitely some improvement here!

I'm still seeing _'source and transform nodesFailed to process remote content'_ for some image files, but it moved past fetching collections where it keeps crashing. It appears to be having the exact same issue but further on in the proccess.

I'm going to fiddle a little with this create-file-node.js when I have time and report back. I'm still in search for a solution.

my-shopify-store $  gatsby develop
success open and validate gatsby-configs — 0.017 s
success load plugins — 0.180 s
success onPreInit — 0.576 s
success delete html and css files from previous builds — 0.011 s
success initialize cache — 0.263 s
success copy gatsby files — 0.055 s
success onPreBootstrap — 0.004 s
⠁ 
gatsby-source-shopify/visuals-by-impulse starting to fetch data from Shopify
⠂ source and transform nodes
gatsby-source-shopify/visuals-by-impulse fetched and processed blogs: 538.650ms

gatsby-source-shopify/visuals-by-impulse fetched and processed productTypes: 537.498ms

gatsby-source-shopify/visuals-by-impulse fetched and processed policies: 549.117ms
Downloading remote files [===] 25/26 0.6 secs 96%
gatsby-source-shopify/visuals-by-impulse fetched and processed collections: 1516.149ms
Downloading remote files [==] 26/26 1.2 secs 100%


gatsby-source-shopify/visuals-by-impulse fetched and processed articles: 2147.283ms
⢀ source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/premade-logo-patriotic-text-logo-1.png
⠄ source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/Blue_588ad53b-3f1a-499d-a349-2586f2df87a5.png
⠁ source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/stream-package-digital-chroma-stream-package-7.png
⠈ source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/Pink_619c0919-2e88-49bc-85e7-11c793f434e3.png
Downloading remote files [] 589/958 100.3 secs 61Downloading remote files [] 590/958 100.3 secs 62Downloading remote files [] 591/958 100.9 secs 62Downloading remote files [] 592/958 101.0 secs 62Downloading remote files [] 593/958 101.0 secs 62Downloading remote files [] 594/958 101.3 secs 62Downloading remote files [] 595/958 101.3 secs 62Downloading remote files [] 596/958 102.5 secs 62Downloading remote files [] 597/958 102.5 secs 62Downloading remote files [] 598/958 102.6 secs 62Downloading remote files [] 599/958 103.3 secs 63Downloading remote files [] 600/958 103.4 secs 63Downloading remote files [] 601/958 103.9 secs 63Downloading remote files [] 602/958 103.9 secs 63Downloading remote files [] 603/958 103.9 secs 63Downloading remote files [] 604/958 104.6 secs 63Downloading remote files [] 605/958 106.9 secs 63Downloading remote files [] 606/958 107.4 secs 63Downloading remote files [] 607/958 107.6 secs 63Downloading remote files [] 608/958 107.6 secs 63Downloading remote files [] 609/958 107.7 secs 64Downloading remote files [] 610/958 107.7 secs 64Downloading remote files [] 611/958 107.9 secs 64Downloading remote files [] 612/958 108.1 secs 64Downloading remote files [] 613/958 108.3 secs 64Downloading remote files [] 614/958 108.3 secs 64Downloading remote files [] 615/958 108.6 secs 64Downloading remote files [] 616/958 108.9 secs 64Downloading remote files [] 617/958 109.3 secs 64Downloading remote files [] 618/958 109.9 secs 65%Failed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/Dark-Pres.png
Downloading remote files [] 619/958 109.9 secs 65
⠂ source and transform nodesFailed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/16-9-Presentation.png
Downloading remote files [=========================-----] 800/958 159.9 secs 84%Failed to process remote content https://cdn.shopify.com/s/files/1/1353/2527/products/16x9-Cyan-Presentation.png
⠄ source and transform nodes

[violy]

I recently open an issue #13073, and Pull Request on #13110 around the Failed to process remote content warning.

Warning, the Failed to process remote content message, is a very basic behaviour, and error handling is not implemented, and — it’s important — don't stop the build process.
But I think some _next_ source and transform actions based on failing nodes, called after in the build process can fails and never end, due to this non handling / breaking error behaviour, never catched.

So, if you add ${e.message} in console.warn(`Failed to process remote content ${task.url}

https://github.com/gatsbyjs/gatsby/blob/7b09d17cf10bf7cf47c58d072029fa0a411f3a20/packages/gatsby-source-filesystem/src/create-remote-file-node.js#L126

what kind of error messages are displayed ?

[OllieJT]

Apologies for the delay. I took your advice @violy and have seen some results that I don't completely understand. My guess would be that there are assets in the store I am working with that are causing problems, but I don't understand that last part at all.

my-shopify-store $  gatsby develop --verbose
verbose 0.752 set gatsby_log_level: "verbose"
verbose 0.755 set gatsby_executing_command: "develop"
verbose 0.755 loading local command from: /Users/olliejt/Documents/GitHub/my-shopify-store/node_modules/gatsby/dist/commands/develop.js
verbose 3.403 running command: develop
success open and validate gatsby-configs — 0.009 s
success load plugins — 0.554 s
success onPreInit — 0.009 s
success initialize cache — 0.005 s
success copy gatsby files — 0.068 s
success onPreBootstrap — 0.002 s
⠁ 
gatsby-source-shopify/visuals-by-impulse starting to fetch data from Shopify
⠂ source and transform nodes
gatsby-source-shopify/visuals-by-impulse fetched and processed productTypes: 534.850ms

gatsby-source-shopify/visuals-by-impulse fetched and processed blogs: 540.133ms

gatsby-source-shopify/visuals-by-impulse fetched and processed policies: 538.546ms
⢀ source and transform nodes
gatsby-source-shopify/visuals-by-impulse fetched and processed collections: 722.112ms
⠠ source and transform nodes
gatsby-source-shopify/visuals-by-impulse fetched and processed articles: 847.476ms
Downloading remote files [==============================] 5/5 0.3 secs 100%

gatsby-source-shopify/visuals-by-impulse fetched and processed products: 4204.719ms

gatsby-source-shopify/visuals-by-impulse finished fetching data from Shopify: 4226.774ms
success source and transform nodes — 4.308 s
warning Multiple node fields resolve to the same GraphQL field `ShopifyArticle.blog` - [`blog`, `blog___NODE`]. Gatsby will use `blog___NODE`.
warning Multiple node fields resolve to the same GraphQL field `ShopifyCollection.products` - [`products`, `products___NODE`]. Gatsby will use `products___NODE`.
warning Multiple node fields resolve to the same GraphQL field `ShopifyProduct.options` - [`options`, `options___NODE`]. Gatsby will use `options___NODE`.
warning Multiple node fields resolve to the same GraphQL field `ShopifyProduct.variants` - [`variants`, `variants___NODE`]. Gatsby will use `variants___NODE`.
success building schema — 0.390 s
success createPages — 0.311 s
success createPagesStatefully — 0.031 s
success onPreExtractQueries — 0.002 s
success update schema — 0.022 s
success extract queries from components — 0.109 s
error UNHANDLED EXCEPTION


  TypeError: Cannot read property 'bitmap' of undefined

  - Potrace.js:1000 Potrace._processLoadedImage
    [my-shopify-store]/[potrace]/lib/Potrace.js:1000:35

  - Potrace.js:1046 Jimp.<anonymous>
    [my-shopify-store]/[potrace]/lib/Potrace.js:1046:14

  - index.js:85 Jimp.throwError
    [my-shopify-store]/[jimp]/index.js:85:44

  - index.js:201 ReadFileContext.callback
    [my-shopify-store]/[jimp]/index.js:201:44

(sharp:13994): GLib-CRITICAL **: 23:21:57.125: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:13994): GLib-CRITICAL **: 23:21:57.125: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:13994): GLib-CRITICAL **: 23:21:57.125: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:13994): GLib-CRITICAL **: 23:21:57.125: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:13994): GLib-CRITICAL **: 23:21:57.126: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:13994): GLib-CRITICAL **: 23:21:57.126: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:13994): GLib-CRITICAL **: 23:21:57.126: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed

(sharp:13994): GLib-CRITICAL **: 23:21:57.126: g_hash_table_lookup: assertion 'hash_table != NULL' failed
Segmentation fault: 11
my-shopify-store $  

[wardpeet]

@OllieJT that's a bit weird, it's a shapr issue. Mind reinstalling your node_modules? yarn --force or rm -rf node_modules.

[iamskok]

Any updates?

[OllieJT]

Hello, apologies for the late reply. I was unable to resolve this issue and had to move on due to time constraints.

@wardpeet I had already tried that a few times to no avail.

On a whim I fired up a starter using this repo: https://github.com/gil--/gatsby-starter-shopifypwa and it has been working since.

I'll close this for now - let me know if you would like it re-opened.

[Zoe7]

I ran into something similar today except I got no error messages whatsoever, it just hung on source and transform nodes after fetched and processed Collection nodes.

Observing on my project a same pattern warning Failed to process remote content, due to requests overload, try to change concurrent: 200 to lower value, like 10 in create-remote-file-node.js of gatsby-source-filesystem plugin.

https://github.com/gatsbyjs/gatsby/blob/d74b3f08c747cec150ebd62638e0c4f3f8c2aa8f/packages/gatsby-source-filesystem/src/create-remote-file-node.js#L89-L103

I hope it will help you.

Tried this and it helped, you can also set a .env variable for it rather than changing the inline value in the node_modules files: GATSBY_CONCURRENT_DOWNLOAD=10.

[AllanPooley]

Hi everyone. I was getting similar error messages (error an error occurred while sourcing data) and found this issue.

The issue in my case was the value I was using for the accessToken when setting up gatsby-source-shopify in gatsby-config.

I had overlooked that the accessToken is the Storefront access token, not you Shopify private app's API Key.

It would be great if you got a more specific error message when this is the case (that the credentials are invalid).

[rlfrahm]

I had the same problem as the OP and @AllanPooley's comment solved it.

Additionally, you need to make products available to the private app's storefront or else you get a similar error.

[Acidic9]

The issue could be that your store simply has 0 products.

Try adding a dummy product and see if that solves it...

[rlfrahm]

The store had like 6k+ products.

[emilymaskin]

The store had like 6k+ products.

I'm having the same issue, and I believe it's specific to large stores. I just created a new dummy store with 1 product and it was smooth sailing.

[willinglee]

I have the same issue too with a large store.

I get this error: UNHANDLED REJECTION Encountered an error trying to infer a GraphQL type for: 'products___NODE'. I tried all the solutions above but I am still getting the error.

[oalbacha]

1. The appName is what you named your app on shopify. It can be found in the url of shopify when you created your app so appName.myshopify.com. This can sometimes get confusing because the app gets named many times on github, shopify, npm...
2. make sure you are providing the access token and not the API Key as @AllanPooley mentioned
   I had the same error because initially I had the API key instead of accessToken and my appName in gatsby-config.js was wrong

[perryraskin]

i followed this tutorial exactly and i'm getting this error:

AggregateError: GraphQLDocumentError: Cannot query field "allShopifyProduct" on type "Query". Did you mean "allShopifyShopPolicy" or "allShopifyCollection"?
      GraphQLDocumentError: Cannot query field "allShopifyProduct" on type "Query". Did you mean "allShopifyShopPolicy" or "allShopifyCollection"?
      GraphQLDocumentError: Cannot query field "allShopifyProduct" on type "Query". Did you mean "allShopifyShopPolicy" or "allShopifyCollection"?
      GraphQLDocumentError: Cannot query field "allShopifyProduct" on type "Query". Did you mean "allShopifyShopPolicy" or "allShopifyCollection"?
      GraphQLDocumentError: Cannot query field "allShopifyProduct" on type "Query". Did you mean "allShopifyShopPolicy" or "allShopifyCollection"?
      GraphQLDocumentError: Cannot query field "allHomemoderns" on type "PRISMIC". Did you mean "allCommons", "allHomes", or "allCategorys"?
      GraphQLDocumentError: Cannot query field "allHomeminimals" on type "PRISMIC". Did you mean "allCommons" or "allHomes"?
      GraphQLDocumentError: Cannot query field "products" on type "ShopifyCollection".
      GraphQLDocumentError: Cannot query field "products" on type "ShopifyCollection".
      GraphQLDocumentError: Cannot query field "shopifyProduct" on type "Query". Did you mean "shopifyShopPolicy", "shopifyCollection", or "allShopifyShopPolicy"?
      GraphQLDocumentError: Unknown argument "products" on field "shopifyCollection" of type "Query".
      GraphQLDocumentError: Cannot query field "products" on type "ShopifyCollection".
      GraphQLDocumentError: Cannot query field "allHomeminimals" on type "PRISMIC". Did you mean "allCommons" or "allHomes"?
      GraphQLDocumentError: Cannot query field "products" on type "ShopifyCollection".

In addition to other similar ones like:

ERROR #85901  GRAPHQL

There was an error in your GraphQL query:

Unknown argument "products" on field "shopifyCollection" of type "Query".

GraphQL request:32:21
31 |            }
32 |            shopifyCollection(products: { elemMatch: { handle: { eq: $handle } } }) {
   |                     ^
33 |                    products {

File: src/templates/product-page.tsx:251:21

Any help really appreciated!

[graviteski]

I was getting this same error and realised that you need to enter your Shopify shop name in the .env files (or gatsby-config) not the app name as mentioned above. Once I did this it built fine for me.

[perryraskin]

@graviteski that's what I did though :/

I even set the app name to be the same name as the store name to be sure

[VladimirCatrici]

@perryraskin faced the same issue, struggled about an hour with that till accidentally found that products were not available on the private app that was created to obtain the access token and play around with storefront API - you know, the section where you can manage items availability on sales channels and apps.