Gatsby: Gatsby contentful build crash all time with space has many data.
Description
I'm trying to build a gatsby site with contentful. All my build working well at the start. When number of pages is increasing I feeling the build is getting slower. And now when the number of pages is large. I see my build crash all the time.
The first thing I got is the node got an issue about memory. so I try to increase memory for node but it got another issue is keeping take too long to build. I usually stop at 99%.
NODE_OPTIONS=--max_old_space_size=4096 gatsby build
success open and validate gatsby-configs - 0.035s
success load plugins - 1.590s
success onPreInit - 0.005s
success delete html and css files from previous builds - 0.015s
success initialize cache - 0.009s
success copy gatsby files - 0.066s
success onPreBootstrap - 0.015s
success createSchemaCustomization - 0.010s
Starting to fetch data from Contentful
Fetching default locale
default locale is : en-US
contentTypes fetched 55
Updated entries 1667
Deleted entries 0
Updated assets 1445
Deleted assets 0
Fetch Contentful data: 22722.827ms
success source and transform nodes - 32.822s
info Writing GraphQL type definitions to schema.gql
success building schema - 80.731s
success createPages - 1.041s
success createPagesStatefully - 0.071s
success onPreExtractQueries - 0.003s
success update schema - 0.338s
success extract queries from components - 2.514s
success write out requires - 0.009s
success write out redirect data - 0.002s
success onPostBootstrap - 0.002s
â €
info bootstrap finished - 121.149 s
â €
success Building production JavaScript and CSS bundles - 19.703s
success Rewriting compilation hashes - 0.005s
[=========================== ] 146.603 s 557/565 99% run queries
Steps to reproduce
It happens all-time with a contentful space that has many data. I keep getting this issue with a blank contentful starter when connecting to this space as well.
Expected result
Build success with small time.
Actual result
I did a lot of research and see it happen in past but I saw the issue fix and they can get a site with 25k pages run < 1 minute. Amazing. But it happens with me now. Crash all the time.
Environment
System:
OS: macOS 10.14.3
CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 12.4.0 - /usr/local/bin/node
npm: 6.9.0 - /usr/local/bin/npm
Languages:
Python: 2.7.17 - /usr/local/bin/python
Browsers:
Chrome: 77.0.3865.90
Firefox: 72.0.2
Safari: 12.0.3
npmPackages:
gatsby: ^2.19.22 => 2.19.22
gatsby-image: ^2.2.41 => 2.2.41
gatsby-plugin-react-helmet: ^3.1.22 => 3.1.22
gatsby-plugin-react-redux: ^1.1.0-0 => 1.1.0-0
gatsby-plugin-sass: ^2.1.29 => 2.1.29
gatsby-plugin-schema-snapshot: ^1.0.0 => 1.0.0
gatsby-plugin-sharp: ^2.4.5 => 2.4.5
gatsby-source-contentful: ^2.1.87 => 2.1.87
gatsby-transformer-remark: ^2.6.53 => 2.6.53
gatsby-transformer-sharp: ^2.3.14 => 2.3.14
npmGlobalPackages:
gatsby-cli: 2.7.50
chuyenlee
All 23 comments
Hi @chuyenlee !
Sorry to hear you're running into an issue. To help us best begin debugging the underlying cause, it is incredibly helpful if you're able to create a minimal reproduction. This is a simplified example of the issue that makes it clear and obvious what the issue is and how we can begin to debug it.
Alternatively, you can provide me access to your repo and a temporary delivery/preview tokens. Then I can take a look. It is really hard to help without seeing any code.
Thanks for using Gatsby! 💜
vladar
on 26 Feb 2020
Hi, @vladar.
Thanks for your quick response.
I invited you to access my repo. Now you can access it and see my contentful config at .env* file.
This issue happens with both run as dev or build.
You can change contentful env to staging to see it working well. master env is had many data.
Thanks.
chuyenlee
on 27 Feb 2020
It segfaulted for me. Does it segfaults for you or just hangs indefinitely in the end of the query running?
vladar
on 27 Feb 2020
Did it work with you ? Can you build with it ?
I getting the build can not finish at run queries step.
Thanks.
chuyenlee
on 27 Feb 2020
hi @vladar. What you got when running my code with my contentful space? Thanks
chuyenlee
on 27 Feb 2020
For me it just segfaults with the following error (during source and transform nodes steps and even with --max_old_space_size=4096):
â ‹ source and transform nodes
<--- Last few GCs --->
[10156:00000294C2DEC680] 113981 ms: Mark-sweep 3848.9 (3932.8) -> 3848.7 (3901.3) MB, 181.1 / 0.0 ms (average mu = 0.893, current mu = 0.000) last resort GC in old space requested
[10156:00000294C2DEC680] 114166 ms: Mark-sweep 3848.7 (3901.3) -> 3848.7 (3901.3) MB, 184.6 / 0.0 ms (average mu = 0.792, current mu = 0.000) last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0000021F8A25C5C1]
Security context: 0x02e523f9e6e9 <JSObject>
1: update [000001CDFF4EB551] [internal/crypto/hash.js:62] [bytecode=0000016DE53B5EB9 offset=106](this=0x03f943a0a761 <Hash map = 0000019B52385A61>,data=0x03f943a0a2e1 <Very long string[258026943]>,encoding=0x01b7813026f1 <undefined>)
2: arguments adaptor frame: 1->2
3: digest(aka digest) [00000055FA0E2D29] [C:\dev\tmp\petinsurancerepo\node_modu...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: 00007FF7B53F832A v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+4506
2: 00007FF7B53D2DB6 node::MakeCallback+4534
3: 00007FF7B53D3730 node_module_register+2032
4: 00007FF7B56EC14E v8::internal::FatalProcessOutOfMemory+846
5: 00007FF7B56EC07F v8::internal::FatalProcessOutOfMemory+639
6: 00007FF7B58D2874 v8::internal::Heap::MaxHeapGrowingFactor+9620
7: 00007FF7B58D0D3B v8::internal::Heap::MaxHeapGrowingFactor+2651
8: 00007FF7B59FAB9B v8::internal::Factory::AllocateRawWithImmortalMap+59
9: 00007FF7B59FD64D v8::internal::Factory::NewRawTwoByteString+77
10: 00007FF7B5748688 v8::internal::Smi::SmiPrint+536
11: 00007FF7B56DF75B v8::internal::StringHasher::UpdateIndex+219
12: 00007FF7B5704F4B v8::String::WriteUtf8+171
13: 00007FF7B5323B3C RSA_meth_get_flags+35820
14: 00007FF7B52C82CF X509_getm_notAfter+43535
15: 00007FF7B52D2591 X509_getm_notAfter+85201
16: 00007FF7B58FEC62 std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Umove+79442
17: 00007FF7B59000ED std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Umove+84701
18: 00007FF7B58FF146 std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Umove+80694
19: 00007FF7B58FF02B std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Umove+80411
20: 0000021F8A25C5C1
But you have [email protected] in your package-lock.json. I'll try it with the latest now.
vladar
on 27 Feb 2020
Hi @vladar.
Thanks. With --max_old_space_size=4096. I got it running forever at run queries step. Hope you can found something.
What do you think about use contentful to manage data when have many pages like that.
Thanks
chuyenlee
on 27 Feb 2020
It looks like a cycle in data somewhere. We had several issues about this in past but I was hoping they're fixed. Will debug it further tomorrow.
Gatsby and Contentful should handle this amount of pages without much trouble. It just looks like a bug (probably an edge case with your content model).
vladar
on 27 Feb 2020
Hi @vladar, we have the same issue, is there any information to provide for helping purposes?
Thanks for caring ;-)
peterpunk
on 27 Feb 2020
@peterpunk Thanks for letting me know! Do you have a segfault or does it just hang for you during query running step? I think having access to your site or a reproduction would be helpful - I can verify my findings this way (if I find anything).
vladar
on 27 Feb 2020
in some machines hangs during build, when we run in gcloud build we get seg fault even with high-cpu, we tested node version 10 and 12,
with 10 we get:
Step #4: success Generating image thumbnails - 10.708s - 12/12 1.12/s
Step #4:
Step #4: <--- Last few GCs --->
Step #4:
Step #4: [17:0x4311140] 132730 ms: Mark-sweep 1374.0 (1423.7) -> 1374.0 (1424.7) MB, 1758.1 / 0.0 ms (average mu = 0.115, current mu = 0.002) allocation failure scavenge might not succeed
Step #4: [17:0x4311140] 134541 ms: Mark-sweep 1374.8 (1424.7) -> 1374.8 (1426.2) MB, 1808.3 / 0.0 ms (average mu = 0.061, current mu = 0.002) allocation failure scavenge might not succeed
Step #4:
Step #4:
Step #4: <--- JS stacktrace --->
Step #4:
Step #4: ==== JS stack trace =========================================
Step #4:
Step #4: 0: ExitFrame [pc: 0x2bd837edbe1d]
Step #4: 1: StubFrame [pc: 0x2bd837edd190]
Step #4: Security context: 0x03c44189e6e1 <JSObject>
Step #4: 2: /* anonymous */(aka /* anonymous */) [0x19723146aa71] [/workspace/node_modules/terser/dist/bundle.min.js:~1] [pc=0x2bd83904f62e](this=0x05d58ac026f1 <undefined>)
Step #4: 3: arguments adaptor frame: 1->0
Step #4: 4: /* anonymous */(aka /* anonymous */) [0x19723146aaf1] [/workspace/node_modules/terser/dist/bundle.min.js:~1] ...
Step #4:
Step #4: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Step #4: 1: 0x8dc1c0 node::Abort() [node]
Step #4: 2: 0x8dc20c [node]
Step #4: 3: 0xad60ae v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
Step #4: 4: 0xad62e4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
Step #4: 5: 0xec3972 [node]
Step #4: 6: 0xec3a78 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
Step #4: 7: 0xecfb52 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
Step #4: 8: 0xed0484 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
Step #4: 9: 0xed30f1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
Step #4: 10: 0xe9c574 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
Step #4: 11: 0x113beae v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
Step #4: 12: 0x2bd837edbe1d
Step #4: Aborted (core dumped)
with 12 we get:
success Generating image thumbnails - 12.346s - 12/12 0.97/s
Killed
npm ERR! code ELIFECYCLE
npm ERR! errno 137
npm ERR! osc-starter@ build: `gatsby build -- --max_old_space_size=8192`
npm ERR! Exit status 137
npm ERR!
npm ERR! Failed at the osc-starter@ build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
in my machine (16gb) the build works but takes ages for just 5 pages:
info Done building in 435.348 sec.
now I'm rolling back to previous versions to see which one works better.
Which kind of access do you need?
Thanks
peterpunk
on 27 Feb 2020
hi @vladar
I think I found the issue.
I think my issue related to this issue https://github.com/gatsbyjs/gatsby/issues/11364
"Not sure if this is a bug or not. And unfortunately I don't really have much time this week to investigate. But I thought I might as well put this up here.
I'm using the gatsby-source-contentful plugin. In my contentful content model I'm having two content types, both with the new Richtext editor. Problems occurs when I have entry links to each other.
To be more clear. Say I have two pages, home and about. If I link from home -> about and from about -> home inside the Richtext editor I get this error.
Like I said before. Not sure if this is a bug.
"
What is the best solution for this case?
Thanksk.
chuyenlee
on 28 Feb 2020
@chuyenlee I thought about this issue at first. Then I tried your code with the latest gatsby and contentful plugin (which were supposed to have a fix for it) and it didn't help. So this can be something else.
But if you want to check yourself - just upgrade to the latest versions of both - gatsby and the contentful-source plugin.
vladar
on 28 Feb 2020
@vladar
I found when 2 pages link together by hyperlink Entry page then we will get this issue, so maybe it's an issue that contentful-source plugin not cover.
Should we make change at this plugin ?
Thanks.
chuyenlee
on 28 Feb 2020
Yeah, that's what I meant about cyclic references. This can be a plugin OR a core issue. Need to figure it out first.
vladar
on 28 Feb 2020
Ok so after removing node_modules and package-lock and upgrading to latest gatsby and gatsby-source-contentful it is not segfaulting anymore but I do see that queries hang on:
[============================] 77.626 s 566/567 100% run queries
@chuyenlee So the issue is that one of your page queries is huge (in pages/healthzone.js). It never finishes for me. If I remove ALL scalar fields from it (and only keep __typename and structural fields) - it runs quickly but still resulting JSON is 3.5MB.
I tried removing parts of the query it was executed eventually, but returned 30MB of data. So I suspect full query returns hundreds of megabytes (maybe more).
Maybe it's because you have cycles in data. But the fact is right now you query all of your data multiple times on different levels of a query. And that's why it hangs.
So what you can do here:
- Temporary remove
pages/healthzone.jsfrom your build - Run gatsby develop
- Open GraphiQL and try to narrow down why this query blows up like this (step by step).
In Gatsby core we will probably add a warning for long-running queries pointing to a component that causes this.
vladar
on 28 Feb 2020
hi @vladar
Thanks for you update. The segfaulting was gone because I remove the cycle reference at my content field already. Because of this issue so I can not use the hyperlink entry URL feature. I must use fix link instead.
How you know the issue from healthzone.js. have any report to see where has a huge query. ?
I will try to improve this page more.
Thanks so much.
chuyenlee
on 28 Feb 2020
@vladar
One other case is i have content type is categories. Thats can have 4 levels. Each category will have a slug field and parent field. So with 4 level i need generate full link for category like level1/level2/level3/level4. What is best practice to generate slug for each category. For now i must query like that :
allContentfulHzCategory {
edges {
node {
id
categoryName
contentName
contentful_id
slug
parentCategory {
id
categoryName
contentName
contentful_id
slug
parentCategory {
id
categoryName
contentName
contentful_id
slug
parentCategory {
id
categoryName
contentName
contentful_id
slug
parentCategory {
id
categoryName
contentName
contentful_id
slug
parentCategory {
id
categoryName
contentName
contentful_id
slug
}
}
}
}
}
}
}
}
Thanks
chuyenlee
on 28 Feb 2020
@vladar question: is there a way to trace the queries being done? so we can see where is getting stuck
peterpunk
on 28 Feb 2020
@peterpunk Unfortunately there is no easy way to do this at the moment. I just used debugger and a bunch of console.log to figure this out.
vladar
on 28 Feb 2020
@vladar no problem, we found that the images were too big and that caused in our case the out of memory issue.
Thanks a lot!
peterpunk
on 28 Feb 2020
Thank you for opening this issue, @chuyenlee
I am closing it for now because we have an actionable takeaway: #21825 but please feel free to reopen this and comment if you would like to continue this discussion. We hope we managed to help and thank you for using Gatsby! 💜
vladar
on 2 Mar 2020
Related issues
timbrandin
·
3Comments
kalinchernev
·
3Comments
dustinhorton
·
3Comments
benstr
·
3Comments
brandonmp
·
3Comments