Gatsby: [gatsby-source-shopify] Timeouts during load; Limit of 250 is too large?

[cj2]

Description

Gatsby Shopify source plugin throws unhandled Timeout exception while building cache. When modifying the GraphQL query to use first: 150 rather than first: 250, the cache is built and Gatsby starts.

Steps to reproduce

1. Create a new Gatsby project
2. Add the Shopify source with API key (our store has 4300 products inclusive of unique variants)
3. Run yarn start

Expected result

Gatsby starts normally, with a fully built Shopify cache.

Actual result

Gatsby fails to start

npm WARN lifecycle The node binary used for scripts is /var/folders/zr/18gw2jpj4xn9lfnl8rktth7m0000gp/T/yarn--1558714436525-0.14726619247761996/node but npm is using /Users/cj/.nvm/versions/node/v12.3.1/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.

> [email protected] develop /Users/cj/Companies/aspenandpine/aspenandpine-gatsby
> gatsby develop

success open and validate gatsby-configs — 0.006 s
success load plugins — 1.127 s
success onPreInit — 0.010 s
success initialize cache — 0.061 s
success copy gatsby files — 0.090 s
success onPreBootstrap — 0.018 s

gatsby-source-shopify/aspen-pine starting to fetch data from Shopify

gatsby-source-shopify/aspen-pine fetched and processed blogs: 1433.211ms

gatsby-source-shopify/aspen-pine fetched and processed productTypes: 1461.054ms

gatsby-source-shopify/aspen-pine fetched and processed policies: 1477.566ms
error (node:47418) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated

gatsby-source-shopify/aspen-pine fetched and processed collections: 3005.577ms


gatsby-source-shopify/aspen-pine fetched and processed articles: 3365.866ms
error 
error an error occured while sourcing data
error - 
  message: Timeout
error query: 
  """

      query GetProducts($first: Int!, $after: String) {
        shop {
          products(first: $first, after: $after) {
            pageInfo {
              hasNextPage
            }
            edges {
              cursor
              node {
                availableForSale
                createdAt
                description
                descriptionHtml
                handle
                id
                images(first: 250) {
                  edges {
                    node {
                      id
                      altText
                      originalSrc
                    }
                  }
                }
                onlineStoreUrl
                options {
                  id
                  name
                  values
                }
                priceRange {
                  minVariantPrice {
                    amount
                    currencyCode
                  }
                  maxVariantPrice {
                    amount
                    currencyCode
                  }
                }
                productType
                publishedAt
                tags
                title
                updatedAt
                variants(first: 250) {
                  edges {
                    node {
                      availableForSale
                      compareAtPrice
                      id
                      image {
                        altText
                        id
                        originalSrc
                      }
                      price
                      selectedOptions {
                        name
                        value
                      }
                      sku
                      title
                      weight
                      weightUnit
                    }
                  }
                }
                vendor
              }
            }
          }
        }
      }

  """
variables: 
  first: 250
  after: null
success source and transform nodes — 19.002 s
warning Multiple node fields resolve to the same GraphQL field `ShopifyCollection.products` - [`products`, `products___NODE`]. Gatsby will use `products___NODE`.
error UNHANDLED REJECTION

Environment

System:
OS: macOS 10.14.4
CPU: x64 Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz
Shell: 5.3 - /bin/zsh
Binaries:
Node: 12.3.1 - ~/.nvm/versions/node/v12.3.1/bin/node
Yarn: 1.16.0 - ~/.nvm/versions/node/v12.3.1/bin/yarn
npm: 6.9.0 - ~/.nvm/versions/node/v12.3.1/bin/npm
Browsers:
Chrome: 74.0.3729.169
Firefox: 64.0.2
Safari: 12.1
npmPackages:
gatsby: ^2.5.7 => 2.5.7
gatsby-image: ^2.1.0 => 2.1.0
gatsby-plugin-manifest: ^2.1.1 => 2.1.1
gatsby-plugin-offline: ^2.1.1 => 2.1.1
gatsby-plugin-react-helmet: ^3.0.12 => 3.0.12
gatsby-plugin-sharp: ^2.0.37 => 2.0.37
gatsby-source-filesystem: ^2.0.37 => 2.0.37
gatsby-source-shopify: latest => 2.0.32
gatsby-transformer-sharp: ^2.1.19 => 2.1.19

[freiksenet]

Hello @cj2!

This looks tricky. It doesn't look like either we or graphql-request library that we use actually adds timeout anywhere. I wonder if there is some default that one can override, because node-fetch mentions OS timeout. Could you try if modifying createClient for gatsby-source-shopify/lib.js by adding timeout: 100000000 as an option helps?

If not, then this becomes even trickier, because we can't always know if a request finishes within a timeout, even if eg query data through pagination.

[mojosef]

Is there a way to filter on the initial build? For example, we only need a particular collection that would obviously be well within the timeout limitation ( we are also getting this error on build ).

[cj2]

timeout: 100000000

Hi @freiksenet ,

Thank you for your help! I made this change and it did not seem to have any effect:

/**
 * Create a Shopify Storefront GraphQL client for the provided name and token.
 */
const createClient = (shopName, accessToken) => new _graphqlRequest.GraphQLClient(`https://${shopName}.myshopify.com/api/graphql`, {
  headers: {
    "X-Shopify-Storefront-Access-Token": accessToken
  },
  timeout: 100000000
});

[freiksenet]

@cj2 Ok, that's disappointing :(

Do you want to make a PR to gatsby-source-shopify that makes amount of data queried for each page configurable? We would really appreciate your help here.

[cj2]

@cj2 Ok, that's disappointing :(

Do you want to make a PR to gatsby-source-shopify that makes amount of data queried for each page configurable? We would really appreciate your help here.

Thanks again. Will see what I can do... I tried to create a PR prior to filing the issue, but the docs were a little confusing. Stay tuned!

[cj2]

Is there a way to filter on the initial build? For example, we only need a particular collection that would obviously be well within the timeout limitation ( we are also getting this error on build ).

@mojosef I submitted a Pull Request to resolve this. We have had consistently reliable results setting the parameter in the PR to 100. Hope this helps!

[freiksenet]

Thank you so much @c2j! Closing.

[kingingcole]

Hello. I do not know if this issue has bee fixed, but I still get a timedout error, even with the paginationSize set to as low as 50. I have also set to 100 and it doesn't work, not even Netlify deployed it.
Any help please?

[brod-dotdev]

Also getting a timeout error but it makes a bit less sense (to me)..

query GetProducts($first: Int!, $after: String) {
 ...
}

UNHANDLED REJECTION Encountered an error trying to infer a GraphQL type for: `products___NODE`. There is no corresponding node with the `id` field matching: "Shopify__Product__Z2lkOi8vc2hvcGlmeS9Qcm9kdWN0LzQ1Mjk1NDU3MDc2NTY=, ...".

To me this suggests that we found the product in one query (because we have the Storefront ID) but in another query or parsing process we couldn't find the Product Node related to that ID?

cc @max-dotdev @b1234567890

[RodrigoPrior]

I'm having the same issue.

This is the limit that works in order to build the graphql against shopify:

    query={graphql`
      query ProductList {
        allShopifyProduct(sort: {fields: [createdAt], order: DESC}, limit: 45) {

Any idea?

[freiksenet]

You can configure pagination size in plugin options. Try if a smaller number works.