Gatsby: [gatsby-source-contentful] - Build fails when entries reference eachother

[Jimmydalecleveland]

Description

When using a content type (on Contentful's site) with a "reference" field that can point to the same type, it's possible to create a maximum call stack error that causes Gatsby to fail to build.

As an example, if you had a "Page" content type that, at any point within, has a possible link to other "Page" types, a sort of circular loop can happen if it ends up pointing back to the origin (e.g. Page A contains a reference to Page B and Page B contains a reference to Page A). This error is thrown:

"gatsby-source-contentful" threw an error while running the sourceNodes lifecycle:

Maximum call stack size exceeded

Subsequent schema errors followed for us, which ended up being red herrings. After finally realizing what the real issue was, I tracked down the origin of this bug to version 2.1.73 (https://www.npmjs.com/package/gatsby-source-contentful/v/2.1.73), as it does not occur in 2.1.72 all the way back to 2.0.46.

Steps to reproduce

In the Contentful UI

1. Create a Content Type with field that is a reference. Here is an example we have:
   

1. Create 2 entries of the previous type and make that reference field point to each other. In our situation we wanted a field on "Page" to be able to reference another "Page" for SEO canonical purposes so the URL would automatically update on changes. This also occurred for us when other content types referenced a "Page", not just a type referencing the same type.

2. Run Gatsby Build and you should get the error shown above.

Expected result

Gatsby should build and the graphql data should be available for the types you created.

Actual result

The build fails with a Maximum call stack size exceeded

Environment

This is mine but others have encountered it on OS X.

System:
OS: Linux 4.19 Ubuntu 18.04.4 LTS (Bionic Beaver)
CPU: (8) x64 Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
Shell: 5.4.2 - /usr/bin/zsh
Binaries:
Node: 12.16.1 - ~/.nvm/versions/node/v12.16.1/bin/node
Yarn: 1.18.0 - ~/.nvm/versions/node/v12.16.1/bin/yarn
npm: 6.13.4 - ~/.nvm/versions/node/v12.16.1/bin/npm

[ascorbic]

@wardpeet @pvdz Could you look at this?

[wardpeet]

Thanks, I'll add it to the contentful refactor list

[pveen2]

We run into the same problem, is there anybody with a fix, even if it dirty?

[axe312ger]

Hey, I am trying to reproduce this issue, but self-referencing entities work fine for me on single and multiple reference fields.

Can somebody provide a content model from Contentful where this issues pops up?

This is mine, you can import it into a new space via the contentful cli import command to see if I match the self-referencing pattern mentioned above (https://github.com/contentful/contentful-cli/tree/master/docs/space/import)

My content model with self referecing entries: contentful-export-self-referencing-entities.json.zip

[digitalbase]

I think this is the same issue as https://github.com/gatsbyjs/gatsby/issues/20279

[Jimmydalecleveland]

Sorry for taking so long to get back to this. Crazy month. I can try to put something together this weekend.

For more information in the meantime: this problem only occurred for us when any chain of entries pointed back on the fields owner (parent entry). For example:
1) Page has a field "canonical" that allows a single reference to another Page
2) Page A - canonical links to Page B (no problems here)
3) Page B - canonical links to Page A (circular because GraphQL is trying to nest the data of each page under it's canonical field back and forth infinitely)

This is a silly example, but there were many ways it could happen more subtly, through content type entries.

[axe312ger]

@Jimmydalecleveland an example to reproduce this bug would be great. I have similar structure and it works well. Gatsby can handle circular references that way.

Are you sure you are not using rich text fields containing references?

[digitalbase]

I am trying to emulate (come up with a test case myself).

How should one solve rich text field cycling references?

[Jimmydalecleveland]

@axe312ger Sorry for leaving you hanging. We are launching a site (using Contentful!) this week, so I've hardly had time for anything. I talked with another teammate who created the original model setup that had the recursion problem about recreating the issue soon.

If you want to (or want me to) close out this ticket so you can move on until we are able to do so, that is fine. I can make a new one when I have an example setup. If you'd prefer to keep it open then I'll post that setup example here.

[axe312ger]

For all of you running into this: We are working on a fix. Unfortunately, its not trivial :(

See #24905

[jamesdbruner]

@Jimmydalecleveland @axe312ger I went ahead and created a minimal reproduction repo, and put together a minimal contentful space to reproduce the issue. I might be a little bit late to the party but it may be helpful to test it there.

Currently, if you clone that repo down, run npm i && npm run dev it will fail with the above-mentioned error.

[Jimmydalecleveland]

Cloned and confirmed the error existed. Thanks @jamesdbruner .

[pvdz]

I just tried the repro and could not repro.

 😶  12:22:10 (master) contentful-circular-ref-issue-23120 $ gatsby build
success open and validate gatsby-configs - 0.043s
success load plugins - 0.689s
success onPreInit - 0.008s
success delete html and css files from previous builds - 0.013s
success initialize cache - 0.007s
success copy gatsby files - 0.023s
success onPreBootstrap - 0.017s
success createSchemaCustomization - 0.010s
Starting to fetch data from Contentful
info Fetching default locale
info default locale is: en-US
info contentTypes fetched 1
info Updated entries 2
info Deleted entries 0
info Updated assets 0
info Deleted assets 0
Fetch Contentful data: 1030.900ms
success source and transform nodes - 1.101s
success building schema - 0.305s
success createPages - 0.002s
success createPagesStatefully - 0.056s
success onPreExtractQueries - 0.003s
success update schema - 0.027s
success extract queries from components - 0.435s
success write out requires - 0.004s
success write out redirect data - 0.002s
success onPostBootstrap - 0.002s
⠀
info bootstrap finished - 5.153s
⠀
success Building production JavaScript and CSS bundles - 5.836s
success Rewriting compilation hashes - 0.004s
success run page queries - 0.019s - 1/1 53.23/s
success Building static HTML for pages - 0.466s - 1/1 2.15/s
success onPostBuild - 0.002s
info Done building in 11.644502038 sec
 😶  12:22:28 (master) contentful-circular-ref-issue-23120 $ gatsby clean; gatsby develop
info Deleting .cache, public
info Successfully deleted directories
success open and validate gatsby-configs - 0.026s
success load plugins - 0.422s
success onPreInit - 0.007s
success initialize cache - 0.013s
success copy gatsby files - 0.027s
success onPreBootstrap - 0.012s
success createSchemaCustomization - 0.007s
Starting to fetch data from Contentful
info Fetching default locale
info default locale is: en-US
info contentTypes fetched 1
info Updated entries 2
info Deleted entries 0
info Updated assets 0
info Deleted assets 0
Fetch Contentful data: 597.973ms
success source and transform nodes - 0.664s
success building schema - 0.242s
success createPages - 0.003s
success createPagesStatefully - 0.058s
success onPreExtractQueries - 0.003s
success update schema - 0.028s
success extract queries from components - 0.387s
success write out requires - 0.030s
success write out redirect data - 0.004s
success onPostBootstrap - 0.002s
⠀
info bootstrap finished - 3.564s
⠀
success run page queries - 0.037s - 2/2 53.82/s
⠀
You can now view contentful-starter-gatsby-blog in the browser.
⠀
  http://localhost:8000/
⠀
View GraphiQL, an in-browser IDE, to explore your site's data and schema
⠀
  http://localhost:8000/___graphql
⠀
Note that the development build is not optimized.
To create a production build, use gatsby build
⠀
success Building development bundle - 3.764s
success Re-building development bundle - 0.060s

[axe312ger]

Thanks @jamesdbruner. Will try you repo with the fix as well :)

[jamesdbruner]

@pvdz That's strange. I've tried it multiple times and had @Jimmydalecleveland clone that repo down and I can only get it to error.

test on  master [!] is 📦 v0.0.1 via ⬢ v12.16.2
➜ npm run dev

> [email protected] dev /Users/jamesbruner/Desktop/test
> gatsby clean && gatsby develop

info Deleting .cache, public
info Successfully deleted directories
success open and validate gatsby-configs - 0.035s
success load plugins - 1.641s
success onPreInit - 0.007s
success initialize cache - 0.011s
success copy gatsby files - 0.077s
success onPreBootstrap - 0.020s
success createSchemaCustomization - 0.007s
Starting to fetch data from Contentful
Fetching default locale
default locale is : en-US
contentTypes fetched 1

 ERROR #11321  PLUGIN

"gatsby-source-contentful" threw an error while running the sourceNodes lifecycle:

Maximum call stack size exceeded



  RangeError: Maximum call stack size exceeded

  - lodash.js:12135 Function.isString
    [test]/[lodash]/lodash.js:12135:22

  - normalize.js:57 fixId
    [test]/[gatsby-source-contentful]/normalize.js:57:10

  - normalize.js:73
    [test]/[gatsby-source-contentful]/normalize.js:73:11

  - lodash.js:13401
    [test]/[lodash]/lodash.js:13401:38

  - lodash.js:4905
    [test]/[lodash]/lodash.js:4905:15

  - lodash.js:2990 baseForOwn
    [test]/[lodash]/lodash.js:2990:24

  - lodash.js:13400 Function.mapValues
    [test]/[lodash]/lodash.js:13400:7

  - normalize.js:70 fixIds
    [test]/[gatsby-source-contentful]/normalize.js:70:28

  - normalize.js:83
    [test]/[gatsby-source-contentful]/normalize.js:83:12

  - lodash.js:13401
    [test]/[lodash]/lodash.js:13401:38

  - lodash.js:4905
    [test]/[lodash]/lodash.js:4905:15

  - lodash.js:2990 baseForOwn
    [test]/[lodash]/lodash.js:2990:24

  - lodash.js:13400 Function.mapValues
    [test]/[lodash]/lodash.js:13400:7

  - normalize.js:70 fixIds
    [test]/[gatsby-source-contentful]/normalize.js:70:28

  - normalize.js:83
    [test]/[gatsby-source-contentful]/normalize.js:83:12

  - lodash.js:13401
    [test]/[lodash]/lodash.js:13401:38

  - From previous event:

  - api-runner-node.js:258 runAPI
    [test]/[gatsby]/dist/utils/api-runner-node.js:258:22

  - api-runner-node.js:375 Promise.catch.decorateEvent.pluginName
    [test]/[gatsby]/dist/utils/api-runner-node.js:375:15

  - From previous event:

  - api-runner-node.js:374
    [test]/[gatsby]/dist/utils/api-runner-node.js:374:12

  - timers.js:456 processImmediate
    internal/timers.js:456:21

  - From previous event:

  - api-runner-node.js:368
    [test]/[gatsby]/dist/utils/api-runner-node.js:368:11

  - From previous event:

  - api-runner-node.js:275 module.exports
    [test]/[gatsby]/dist/utils/api-runner-node.js:275:12

  - source-nodes.js:84 Object._default [as default]
    [test]/[gatsby]/dist/utils/source-nodes.js:84:36

  - index.js:386 module.exports
    [test]/[gatsby]/dist/bootstrap/index.js:386:49

  - develop-process.js:433 async module.exports
    [test]/[gatsby]/dist/commands/develop-process.js:433:7


warn The gatsby-source-contentful plugin has generated no Gatsby nodes. Do you need it?
success source and transform nodes - 0.492s
success building schema - 0.221s
success createPages - 0.003s
success createPagesStatefully - 0.062s
success onPreExtractQueries - 0.003s
success update schema - 0.026s
success extract queries from components - 0.430s
success write out requires - 0.025s
success write out redirect data - 0.002s
success onPostBootstrap - 0.002s
⠀
info bootstrap finished - 5.829s
⠀
success run page queries - 0.034s - 2/2 58.98/s
⠀
You can now view contentful-starter-gatsby-blog in the browser.
⠀
  http://localhost:8000/
⠀
View GraphiQL, an in-browser IDE, to explore your site's data and schema
⠀
  http://localhost:8000/___graphql
⠀
Note that the development build is not optimized.
To create a production build, use gatsby build
⠀
success Building development bundle - 4.635s

I also wanted to add screenshots of contentful in case you were curious about how that was set up and hopefully, this illustrates the issue for anyone who's not actively aware that stumbles on this issue.

[Jimmydalecleveland]

@pvdz I notice you are using gatsby to run commands, rather than what is installed in the repo by using yarn or npm. Is it possible you have an older gatsby installed globally? Could be newer as well, actually.

[pvdz]

Reasonable;

$ gatsby --version
Gatsby CLI version: 2.12.15
[email protected]
$ npm --version
6.4.1
$ node --version
v10.13.0
$ yarn --version
1.21.0

$ yarn list | grep contentful
├─ @contentful/[email protected]
├─ [email protected]
├─ [email protected]
│  ├─ contentful-batch-libs@^9.0.1
│  ├─ contentful-management@^5.12.1
├─ [email protected]
│  ├─ contentful-sdk-core@^6.4.0
├─ [email protected]
├─ [email protected]
├─ [email protected]
│  ├─ contentful-resolve-response@^1.1.4
│  ├─ contentful-sdk-core@^6.4.5
├─ [email protected]
│  ├─ @contentful/rich-text-types@^13.4.0
│  ├─ contentful@^7.14.4

(Our node minver is 10.13 so that's my default, to make sure the things we change keeps working)

Can you try gatsby build / develop just in case that changes anything?

Fwiw, I tried node_modules/.bin/gatsby build and npm run build and npm run dev with the same result. I also tried gatsby build on node 12, same result.

[Jimmydalecleveland]

I did actually, and updated my gatsby-cli global version and got the same errors

[Jimmydalecleveland]

The package lock in the reproduction repo shows 2.1.73 for the gatsby-source-contentful version, and your yarn list shows 2.3.15 installed. I feel like that shouldn't matter due to the local scope of node_modules in the repo, but is it possible it's using your global version?

[pvdz]

The only thing global on my system are gatsby-cli, gatsby-dev-cli, and yarn. I just did a yarn on that project. The package.json has "gatsby-source-contentful": "^2.1.73",. If you wanted to actually lock it, drop the ^, which tells yarn and npm to use that version specifically.

I see now that there's a package-lock.json, which yarn ignores, so that's why. To be safe in the future, if you want to lock package versions in a project, drop the carets :)

Anyways. After npm install the build indeed crashes.

From there I did some testing. The gatsby-source-contentful is not the culprit here.

gatsby@^2.23 pass
gatsby@^2.22 pass
gatsby@^2.22.0 pass
[email protected] fail (this was where you locked to)
[email protected] fail
[email protected] pass
[email protected] fail
[email protected] fail
[email protected] pass

So something that got fixed between .28 and .29 (that is sooo last month :p).
28: https://github.com/gatsbyjs/gatsby/commit/00ea4063a52ceffe5134639b4a2adb2e9d6f98fe
29: https://github.com/gatsbyjs/gatsby/commit/de4918292b3957ec309912eebb8260408164a310
(Range of commits: https://github.com/gatsbyjs/gatsby/commits/master?before=bfbe02a59008bbf0e006b93b12f9921f64a13747+551 )

I'll be honest; not sure what commit in that range was responsible for the fix. Possibly related to a dependency (yarn.lock) update.

---

Either way, it seems to me like the fix is to upgrade your gatsby version to resolve this problem. Please try that in your real world case. I'm going to close this issue as I'm confident that the repro is fixed by a bumping gatsby to 2.21.29 or beyond. If this doesn't fix your real world case, please re-open with a new repro case.

[Jimmydalecleveland]

@pvdz This reproduction is a clone of Contentful's starter blog just to reproduce the issue. So we aren't intending to lock or not lock anything. That starter comes with a package-lock.json because it was set up with npm.

That's cool you narrowed it down to being fixed by a Gatsby update. I suppose the Contentful starter blog should be updated to use that then.

[Jimmydalecleveland]

I ran an update on that repro for gatsby, which bumped it to 2.23.6, and did the same with my global gatsby install just to be sure. Both failed the same way.

I'll get some people who haven't had any of these packages installed on their machine to run it before bugging you again. Thanks for looking at it.

[axe312ger]

The starter Jimmy was using was pretty much out of date. I upgrade the dependencies here: https://github.com/contentful/starter-gatsby-blog/pull/126

So if you run into this, ensure you run the latest versions of gatsby & gatsby-source-contentful before reopening this. Thanks

[iamaladdin]

@pvdz That's strange. I've tried it multiple times and had @Jimmydalecleveland clone that repo down and I can only get it to error.

test on  master [!] is 📦 v0.0.1 via ⬢ v12.16.2
➜ npm run dev

> [email protected] dev /Users/jamesbruner/Desktop/test
> gatsby clean && gatsby develop

info Deleting .cache, public
info Successfully deleted directories
success open and validate gatsby-configs - 0.035s
success load plugins - 1.641s
success onPreInit - 0.007s
success initialize cache - 0.011s
success copy gatsby files - 0.077s
success onPreBootstrap - 0.020s
success createSchemaCustomization - 0.007s
Starting to fetch data from Contentful
Fetching default locale
default locale is : en-US
contentTypes fetched 1

 ERROR #11321  PLUGIN

"gatsby-source-contentful" threw an error while running the sourceNodes lifecycle:

Maximum call stack size exceeded



  RangeError: Maximum call stack size exceeded

  - lodash.js:12135 Function.isString
    [test]/[lodash]/lodash.js:12135:22

  - normalize.js:57 fixId
    [test]/[gatsby-source-contentful]/normalize.js:57:10

  - normalize.js:73
    [test]/[gatsby-source-contentful]/normalize.js:73:11

  - lodash.js:13401
    [test]/[lodash]/lodash.js:13401:38

  - lodash.js:4905
    [test]/[lodash]/lodash.js:4905:15

  - lodash.js:2990 baseForOwn
    [test]/[lodash]/lodash.js:2990:24

  - lodash.js:13400 Function.mapValues
    [test]/[lodash]/lodash.js:13400:7

  - normalize.js:70 fixIds
    [test]/[gatsby-source-contentful]/normalize.js:70:28

  - normalize.js:83
    [test]/[gatsby-source-contentful]/normalize.js:83:12

  - lodash.js:13401
    [test]/[lodash]/lodash.js:13401:38

  - lodash.js:4905
    [test]/[lodash]/lodash.js:4905:15

  - lodash.js:2990 baseForOwn
    [test]/[lodash]/lodash.js:2990:24

  - lodash.js:13400 Function.mapValues
    [test]/[lodash]/lodash.js:13400:7

  - normalize.js:70 fixIds
    [test]/[gatsby-source-contentful]/normalize.js:70:28

  - normalize.js:83
    [test]/[gatsby-source-contentful]/normalize.js:83:12

  - lodash.js:13401
    [test]/[lodash]/lodash.js:13401:38

  - From previous event:

  - api-runner-node.js:258 runAPI
    [test]/[gatsby]/dist/utils/api-runner-node.js:258:22

  - api-runner-node.js:375 Promise.catch.decorateEvent.pluginName
    [test]/[gatsby]/dist/utils/api-runner-node.js:375:15

  - From previous event:

  - api-runner-node.js:374
    [test]/[gatsby]/dist/utils/api-runner-node.js:374:12

  - timers.js:456 processImmediate
    internal/timers.js:456:21

  - From previous event:

  - api-runner-node.js:368
    [test]/[gatsby]/dist/utils/api-runner-node.js:368:11

  - From previous event:

  - api-runner-node.js:275 module.exports
    [test]/[gatsby]/dist/utils/api-runner-node.js:275:12

  - source-nodes.js:84 Object._default [as default]
    [test]/[gatsby]/dist/utils/source-nodes.js:84:36

  - index.js:386 module.exports
    [test]/[gatsby]/dist/bootstrap/index.js:386:49

  - develop-process.js:433 async module.exports
    [test]/[gatsby]/dist/commands/develop-process.js:433:7


warn The gatsby-source-contentful plugin has generated no Gatsby nodes. Do you need it?
success source and transform nodes - 0.492s
success building schema - 0.221s
success createPages - 0.003s
success createPagesStatefully - 0.062s
success onPreExtractQueries - 0.003s
success update schema - 0.026s
success extract queries from components - 0.430s
success write out requires - 0.025s
success write out redirect data - 0.002s
success onPostBootstrap - 0.002s
⠀
info bootstrap finished - 5.829s
⠀
success run page queries - 0.034s - 2/2 58.98/s
⠀
You can now view contentful-starter-gatsby-blog in the browser.
⠀
  http://localhost:8000/
⠀
View GraphiQL, an in-browser IDE, to explore your site's data and schema
⠀
  http://localhost:8000/___graphql
⠀
Note that the development build is not optimized.
To create a production build, use gatsby build
⠀
success Building development bundle - 4.635s

I also wanted to add screenshots of contentful in case you were curious about how that was set up and hopefully, this illustrates the issue for anyone who's not actively aware that stumbles on this issue.

Hello , @jamesdbruner add this to your gatsby-config.js and make sure that your .env file matches path: .env .

require("dotenv").config({
  path: `.env`,
});