React-snap: Navigation Timeout Exceeded with waitUntil: networkidle*

[cascornelissen]

I've been pulling my hair out on this for a couple of hours now and I'm hoping someone here knows what the problem might be in this case.

I'm running into the following error when executing react-snap on our Jenkins environment, which runs Debian 8 (it runs correctly on my local macOS High Sierra machine)...

🔥  / Error: Navigation Timeout Exceeded: 30000ms exceeded
    at Promise.then (/home/jenkins/jobs/workspace/Freshheads Website/build/node_modules/puppeteer/lib/NavigatorWatcher.js:73:21)
    at <anonymous>

I've tried playing with pretty much every setting available in react-snap but I couldn't get it to work so I started debugging. Turns out changing waitUntil on this line to either load or domcontentloaded resolves my issue.

The strangest thing is that networkidle* should work correctly in my mind: I've enabled logging for Puppeteer by passing DEBUG=puppeteer:session to the command that starts react-snap (output is available right here: log.txt) and it just hangs for about 20 - 25 seconds when the entire output is printed which should mean there is no network traffic going on, right?

My question is; would it be possible to add an option for the waitUntil variable to react-snap or doesn't that work/output the correct HTML?

[stereobooster]

Turns out changing waitUntil on this line to either load or domcontentloaded resolves my issue.

This doesn't solve issue in general it simply masks the problem, though it can work in your case. Issue with timeout is known https://github.com/stereobooster/react-snap/issues/131. It was reported in puppeteer, but it still reported by users from time to time

[cascornelissen]

@stereobooster, thanks for the quick reply! I noticed #131 but since the advice over there is to use 1.10 while the current version is 1.17.0 already I was expecting that to be solved.

I'll try downgrading to 1.10 tomorrow/friday, if that doesn't fix it I'll try and debug some more based on https://github.com/GoogleChrome/puppeteer/issues/1908#issuecomment-380308269.

In the end the load value seems to work correctly for me. Would you accept a PR that adds a waitUntil option?

[stereobooster]

You can use this one https://github.com/peterbe/minimalcss/pull/199/files

[cascornelissen]

@stereobooster, I've done some more testing using the tracker you've linked to in the previous comment.

I've modified it a little bit to make it easier to debug my case:

const createTracker = page => {
    const requests = new Set();
    const onStarted = request => {
        console.log('started', request._url);
        return requests.add(request);
    };
    const onFinished = request => {
        console.log('finished', request._url);
        return requests.delete(request);
    };
    const onFailed = request => {
        console.log('FAILED', request._url);
        return requests.delete(request);
    };
    page.on('request', onStarted);
    page.on('requestfinished', onFinished);
    page.on('requestfailed', onFailed);
    return {
        urls: () => Array.from(requests).map(r => r.url()),
        dispose: () => {
            console.log('requests', requests);

            page.removeListener('request', onStarted);
            page.removeListener('requestfinished', onFinished);
            page.removeListener('requestfailed', onFailed);
        }
    };
};

Which outputs the following: every request starts and finishes (in about 3s) and not a single one fails and I'm still running into the 30s timeout (the last 27s it just hangs).

started http://localhost:45678/
finished http://localhost:45678/
started http://localhost:45678/app.4be508b9885babb8ef75.css
started http://localhost:45678/homerun.5e04a4a0f278c718e2bb.css
started http://localhost:45678/lazysizes.3af867507f1aa122bd24.js
started http://localhost:45678/app.3af867507f1aa122bd24.js
started http://localhost:45678/homerun.3af867507f1aa122bd24.js
finished http://localhost:45678/app.4be508b9885babb8ef75.css
finished http://localhost:45678/lazysizes.3af867507f1aa122bd24.js
finished http://localhost:45678/homerun.3af867507f1aa122bd24.js
finished http://localhost:45678/homerun.5e04a4a0f278c718e2bb.css
finished http://localhost:45678/app.3af867507f1aa122bd24.js
started http://localhost:45678/vendors-home-view.02d944751c4d70716f2a.bundle.js
started http://localhost:45678/home-view.3a19da2afcb08790bdeb.bundle.js
started http://localhost:45678/fonts/neutrif-studio-regular.woff2
started http://localhost:45678/fonts/proxima-nova-extra-bold.woff2
finished http://localhost:45678/home-view.3a19da2afcb08790bdeb.bundle.js
finished http://localhost:45678/fonts/neutrif-studio-regular.woff2
finished http://localhost:45678/fonts/proxima-nova-extra-bold.woff2
finished http://localhost:45678/vendors-home-view.02d944751c4d70716f2a.bundle.js
started http://localhost:45678/fonts/proxima-nova-extra-bold.typeface.json
started http://localhost:45678/fonts/neutrif-studio-regular.typeface.json
finished http://localhost:45678/fonts/proxima-nova-extra-bold.typeface.json
started http://localhost:45678/images/cases/roadguard/blob.jpg
started http://localhost:45678/images/cases/interpolis/blob.jpg
started http://localhost:45678/images/cases/quiet/blob.jpg
started http://localhost:45678/images/cases/viia/blob.jpg
started http://localhost:45678/images/cases/ggz/blob.jpg
finished http://localhost:45678/fonts/neutrif-studio-regular.typeface.json
finished http://localhost:45678/images/cases/quiet/blob.jpg
finished http://localhost:45678/images/cases/ggz/blob.jpg
finished http://localhost:45678/images/cases/viia/blob.jpg
finished http://localhost:45678/images/cases/roadguard/blob.jpg
finished http://localhost:45678/images/cases/interpolis/blob.jpg

You have been evading my question about an option for the waitUntil value though, is that something you're not interested in?

[cascornelissen]

Oh, and @stereobooster, FYI: locking react-snap at 1.10.0 does resolve my problem. I'd love to help you get this fixed/improved in an upcoming release. Let me know if there's anything else I can provide to resolve the issue.

The question about the waitUntil option still stands, though.

[stereobooster]

FYI: locking react-snap at 1.10.0 does resolve my problem.

Can you also say which version of puppeteer do you use (based on package.lock.json or yarn.lock)

[cascornelissen]

└─┬ [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  └── [email protected]

[stereobooster]

This is the thing, in puppeteer v1.0 something is broken. Issue was reported, but closed. We need to reopen the issue and provide reproducible example

[cascornelissen]

Do you have any issue numbers? I might be able to build a simple docker image showing the problem if the developers over at puppeteer are interested.

[stereobooster]

This is the issue https://github.com/GoogleChrome/puppeteer/issues/1908#issuecomment-390214976

[stereobooster]

I have another idea, if we will add tracker (to check that all requests are finished) we can omit false positive Timeout Errors, downside of this approach is that prerendering will take longer time.

[stereobooster]

Related https://github.com/peterbe/minimalcss/pull/248/files#diff-f712f2169bd3dca93334576e35acff9f

[stereobooster]

Closed in favour of https://github.com/stereobooster/react-snap/issues/240

You have been evading my question about an option for the waitUntil value though, is that something you're not interested in?

The problem here, this will mask issue but not solve it. By adding waitUntil I believe some people will start to use without realising, there is something broken down the line.