Gatsby: [Question] Timeout on Netlify build pipeline possibly caused by regression?

Created on 27 Oct 2018 · 14Comments · Source: gatsbyjs/gatsby

Description

Hi, I'm running into a nasty timeout bug on Netlify build pipeline. Site is small enough (~10 pages) that it's definitely not an issue with actual build time.

Some background: My site was initally build with Gatsby v2-beta38. Upgrade and migrating to official v2 result in timeout. (More info on the exact buggy version below)

What I've tried:

Local Gatsby build command works fine.
Running Netlify build-image test-build script does not result in timeout
Manually start the image and run build command also works fine
Only happen with Gatsby. Test with other static site generator indicate that this is not a problem with Netlify.
Every build debugging advices from Netlify: clearing build cache, delete public/.cache to force fresh build, etc.

What I found out:

I've confirm that this happens after a pump in Gatsby version and done a bunch of test builds. Gatsby 2.0.0-beta67 is the suspect. 66 <-> 67
Netlify support suggest that the timeout might be cause by long running process or some kind of watcher not being able to exit successfully. The build log indicate that the build process itself is not erroneous.

10:43:29 PM: success Building production JavaScript and CSS bundles — 103.920 s
10:43:38 PM: success Building static HTML for pages — 9.583 s — 10/10 10.28 pages/second
10:43:39 PM: info Done building in 117.878 sec
10:43:39 PM: Done in 118.11s.
10:43:39 PM: Caching artifacts
10:43:39 PM: Started saving node modules
10:43:39 PM: Finished saving node modules
10:43:39 PM: Started saving yarn cache
10:43:39 PM: Finished saving yarn cache
10:43:39 PM: Started saving pip cache
10:43:39 PM: Finished saving pip cache
10:43:39 PM: Started saving emacs cask dependencies
10:43:39 PM: Finished saving emacs cask dependencies
10:43:39 PM: Started saving maven dependencies
10:43:39 PM: Finished saving maven dependencies
10:43:39 PM: Started saving boot dependencies
10:43:39 PM: Finished saving boot dependencies
10:43:39 PM: Started saving go dependencies
10:43:39 PM: Finished saving go dependencies
11:10:38 PM: Build exceeded maximum allowed runtime

Deploying a fresh gatsby-starter-netlify-cms, which uses v2.0.0+, does not result in timeout. This indicates that this bug is caused by a particular Gatsby API or the by the way I use it.

Steps to reproduce

This is marked as a question because so far I've been unable to reproduce it with a fresh repo. Netlify does not allow cancelling build (and thus force you to wait for timeout, which happen ~30min), so it took me a nightmarish amount of time for me to get this far. Any lead is appreciated.

Expected result

Build succeed without timeout.

What should happen?

Actual result

Build timeout.

Environment

System:
    OS: Linux 4.15 Ubuntu 16.04.5 LTS (Xenial Xerus)
    CPU: x64 Intel(R) Core(TM) i3-7100U CPU @ 2.40GHz
    Shell: 4.3.48 - /bin/bash
  Binaries:
    Node: 8.11.3 - ~/.nvm/versions/node/v8.11.3/bin/node
    Yarn: 1.10.1 - /usr/bin/yarn
    npm: 5.6.0 - ~/.nvm/versions/node/v8.11.3/bin/npm
  npmPackages:
    gatsby: 2.0.0-beta.66 => 2.0.0-beta.66
    gatsby-plugin-less: ^2.0.5 => 2.0.5
    gatsby-plugin-netlify: ^2.0.0 => 2.0.0
    gatsby-plugin-netlify-cms: ^3.0.3 => 3.0.3
    gatsby-plugin-react-helmet: ^3.0.0 => 3.0.0
    gatsby-plugin-remove-trailing-slashes: ^2.0.0 => 2.0.0
    gatsby-source-filesystem: ^2.0.1 => 2.0.1
    gatsby-transformer-remark: ^2.1.2 => 2.1.2

Source

hrisng

Most helpful comment

Add this on gatsby-node.js

const ChildProcess = require('child_process');

exports.onPostBuild = () => {
  ChildProcess.execSync("ps aux | grep jest | grep -v grep | awk '{print $2}' | xargs kill");
};

rafaellincoln on 29 Mar 2019

👍6 ❤2

All 14 comments

Can you try upgrading gatsby to the latest version? E.g. npm install gatsby@latest.

You're 60-70 versions behind right now ;-)

KyleAMathews on 27 Oct 2018

Hi, I was using the relatively new version (2.0.31), but reverting back for testing purpose.
I just found out that beta.67 might not be the one that is causing issue (I forget to clear build cache between test build). I'll do redo the test builds and report back later.

hrisng on 27 Oct 2018

@KyleAMathews Done a bunch of test builds again and confirm this problem come with beta.67. Gatsby version up til beta.66 works fine. And yes, I've tested with the latest version also.
https://github.com/gatsbyjs/gatsby/blob/5749232c29c6d246eea73dbff82909b79903b004/packages/gatsby/src/internal-plugins/query-runner/page-query-runner.js#L68
^ My suspect is this line, but honestly I'm in the dark here. The fact that it only happen on Netlify build pipeline bewildered me. The log does not have anything useful either, and the only useful advice from Netlify is to look for long running process / watcher or maybe some kind of loop.

Really appreciate if anyone can give that particular release a look (compare link) and help me narrow down what could have gone wrong. Thanks a bunch!

hrisng on 27 Oct 2018

Perhaps try deleting different parts of your code and see when it starts working? That seems like the fastest way to narrow down what's causing this.

KyleAMathews on 27 Oct 2018

I'll definitely try debugging more when I have time. Netlify not allowing to cancel build has slowing the process to a crawl though. This has been going on for weeks. I've resorted to build locally and manually upload to Netlify for now.

In the meantime, please keep this open for anyone who might have a lead or facing similar problem.

hrisng on 27 Oct 2018

This sounds specific to your code as I don't recall any one else reporting it so e.g. delete parts of your gatsby-node.js and see what happens.

KyleAMathews on 27 Oct 2018

👍1

E.g. check if you're returning a promise and then don't resolve it as then Gatsby would wait forever for it to finish.

One thing we could do in core is watch for very long running plugins and then warn if after e.g. 10 minutes, a plugin is still working.

KyleAMathews on 27 Oct 2018

👍1

I think I found out why Netlify builds timeout. https://github.com/gatsbyjs/gatsby/issues/7527#issue-352672559
I've confirmed that I have a dangling jest-worker process after build. Unlike the issue above though, the entire code base does not have setInterval nor any use of componentWillMount. However, some dependencies might have been doing something weird. gatsby-node, gatsby-ssr, gatsby-browser do not seem to be the cause of problem. Beta.67 introduces something that switch on the dormant problem. While I believe this is probably a user's error, this behaviour is potentially frustrating / dangerous. It might need to be documented somewhere,

@KyleAMathews Is there any other symptoms / suspicious code or pattern that I need to look out for? Thanks for the tips yesterday btw!
One more question, it's mentioned here https://github.com/gatsbyjs/gatsby/issues/7527#issuecomment-427477831 that This can prevent netlify builds from finishing if you use a version of gatsby before this was fixed. Is this something that you guys are working on? Or is it already fixed in the latest version?

hrisng on 28 Oct 2018

Found it! It was Firebase.

Firebase database init in my api module was calling some setInterval under the hood even without making any query at build time. The workaround is to check for window to make sure it's only init in browser and leave a dummy db in the the module otherwise. Or move all firebase config and init code into some componentDidMount hook.

Firebase Admin SDK seems to be not affected by this, as tested with gatsby-source-firebase. Still this behaviour is super frustrating and should warned by docs.

hrisng on 28 Oct 2018

👍2

Hey @harrisnguyen94 i think I am having the same issue. A dangling jest worker after build. How did you track down what was causing it?

isaac-martin on 10 Dec 2018

@isaac-martin I was suspicious of two things: promise that was not resolved in gatsby-node and setnterval call that was not cleared. Since my own code base itself had neither of those, I did a grep for the latter in node_modules, which led me to firebase and the way I used it.

hrisng on 11 Dec 2018

For other people stumbling upon this: aws-amplify causes the same issue.

Amplify.configure(amplifyConfig);

sebastiangraef on 14 Dec 2018

🎉1 👍1

Add this on gatsby-node.js

const ChildProcess = require('child_process');

exports.onPostBuild = () => {
  ChildProcess.execSync("ps aux | grep jest | grep -v grep | awk '{print $2}' | xargs kill");
};

rafaellincoln on 29 Mar 2019

👍6 ❤2