Puppeteer: [umbrella] Fix Request Interception
Umbrella bug for all request interception issues in Puppeteer.
This mostly depends on upstream Chromium bug: https://crbug.com/899386
Description
DevTools Protocol emits two events for every intercepted request:
Puppeteer extracts different pieces of information from these events. However, since interceptionId and requestId aren't matching, Puppeteer uses heuristics to match these events together. This is necessary to provide a consistent API to our clients.
Unfortunately, these heuristics aren't reliable and break under different corner cases: #1664, #2848, #3030, #3369 and so on.
Mitigation
We weren't able to fix this upstream up until today due to the network stack refactoring in Chromium. Now, with Network Service getting gradually adopted, we can fix this upstream.
The plan is:
- Enable Network Service by default to Puppeteer users
- Fix network instrumentation in DevTools protocol for the Network Service and migrate onto it. (see https://crbug.com/899386).
aslushnikov
All 52 comments
@aslushnikov Do you think that this issue could cause this:
(node:3397) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 524372)
(node:3397) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'redirectResponse' of undefined
at NetworkManager._onRequest (/home/user/project/node_modules/puppeteer/lib/NetworkManager.js:197:15)
at NetworkManager._onRequestIntercepted (/home/user/project/node_modules/puppeteer/lib/NetworkManager.js:183:12)
at emitOne (events.js:116:13)
at CDPSession.emit (events.js:211:7)
at CDPSession._onMessage (/home/user/project/node_modules/puppeteer/lib/Connection.js:216:12)
at Connection._onMessage (/home/user/project/node_modules/puppeteer/lib/Connection.js:99:19)
at WebSocketTransport._ws.addEventListener.event (/home/user/project/node_modules/puppeteer/lib/WebSocketTransport.js:41:24)
at WebSocket.onMessage (/home/user/project/node_modules/ws/lib/event-target.js:120:16)
at emitOne (events.js:116:13)
at WebSocket.emit (events.js:211:7)
I can't reproduce it but it is happening occasionally. I've looked in to the NetworkManager.js, I suppose the reason was that the requestWillBeSentEvent was not in the Map (NetworkManager.js:182). Do you think that this bug is related or should I open a new issue for this?
yemreinci
on 31 Oct 2018
@yemreinci It doesn't look related to this particular interception issue, can you please file separately?
I have some thoughts regarding it, but yet to reproduce in tests.
aslushnikov
on 1 Nov 2018
For the record: #3421 has a lot of good repros.
aslushnikov
on 1 Nov 2018
@aslushnikov Out of curiosity, is there a rough ETA for this? eg weeks or months away?
sradu
on 12 Nov 2018
@sradu we had a few attempts already, and it always gets quite deep into the Blink internals. I'd say we'll be lucky to land this by the end of the year.
aslushnikov
on 13 Nov 2018
Got it! This is helpful!
sradu
on 13 Nov 2018
In absence of an upstream fix for this, is there a way to use networkidle2 OR a specific time period? We prefer a complete DOM, but are OK to continue after 30 seconds regardless, but it seems there's no way to do this.
jared-w-smith
on 15 Nov 2018
@jared-w-smith you can just race the page.goto navigation with a timeout promise:
await Promise.race([
page.goto(url, {waitUntil: 'networkidle2'}),
new Promise(x => setTimeout(x, 30000)),
]);
Is this what you want?
aslushnikov
on 15 Nov 2018
@aslushnikov Yes, it looks like that should do the trick. Thank!
jared-w-smith
on 15 Nov 2018
@aslushnikov Hypothetically, how would you retrieve the response in this scenario? Is it possible?
To achieve this for now I'm keeping track of navigation request myself in the interceptor because ifgoto throw a timeout exception I won't have it.
panthony
on 15 Nov 2018
@panthony you can track navigation requests with the "request" event and isNavigationRequest flag.
page.on('request', request => {
if (request.isNavigationRequest()) {
// ... save request for later, use request.response() to get response ...
}
});
@aslushnikov While your code...
await Promise.race([
page.goto(url, {waitUntil: 'networkidle2'}),
new Promise(x => setTimeout(x, 30000)),
]);
... will fulfill the promise upon networkidle2 or 30 seconds, the full process won't complete until the 30 seconds is completed. It seems this needs a clearTimeout when the promise is fulfilled, but I'm not sure how best to implement.
jared-w-smith
on 15 Nov 2018
will fulfill the promise upon networkidle2 or 30 seconds, the full process won't complete until the 30 seconds is completed.
IIUC you mean the node.js process. In this case, you'll need to unref the timeout:
await Promise.race([
page.goto(url, {waitUntil: 'networkidle2'}),
new Promise(x => setTimeout(x, 30000).unref()),
]);
@aslushnikov Thank you. This did the trick. I now have to work through some downstream issues if the timeout is reached and the page really isn't available, but this seems better than having numerous incorrect timeouts due to this bug.
jared-w-smith
on 16 Nov 2018
Try downgrade ppt version to 1.6.2. It work for me.
quocthinh212
on 29 Nov 2018
Yes, 1.6.2 is the last version that worked here as well.
sradu
on 29 Nov 2018
For the record, this also causes resources injected using document.write to never load. Minimal test case: https://gist.github.com/kdzwinel/6d887e1237353698d9aebefd013b8302 Downgrading to 1.6.2 works.
kdzwinel
on 9 Jan 2019
We tested the commit bd9bc9c against request interception hang issue as mentioned in GoogleChrome/puppeteer#2848 and it solved the problem. We'd appreciate much if change is shipped in the next release of puppeteer :-) Thanks for the great work! :tada:
Testers: @tli5 @andrewiggins
tli5
on 10 Jan 2019
@aslushnikov just a friendly 2019 checkin to ask about a rough ETA for when this could land. Thank you!
sradu
on 16 Jan 2019
@sradu we're working on it. Still no ETA.
aslushnikov
on 22 Jan 2019
The previous issues with RequestInterception seem to have been addressed in puppeteer@next. Is this correct? Are there still known issues there? Also, were these fixes included in 1.12.2, and if not, when would they be expected in the stable package?
jared-w-smith
on 22 Feb 2019
Nothing has been done yet to address the underlying request id matching problem. @aslushnikov has something been changed/fixed?
josepharhar
on 22 Feb 2019
FWIW, for my integration testing solution, as an alternative to using puppeteer request interception, I ended up creating a local proxy server with Node.js using http-mitm-proxy and then recording/loading fixtures using nock. Nock intercepts requests by hijacking the Node.js core modules http and https, which the proxy uses.
Requests are forced through the proxy server using the Chromum flags --proxy-server and --ignore-certificate-errors. The local Node.js proxy instead does https cert validation.
freewil
on 22 Feb 2019
I'm having a similar problem with puppeteer-core and the latest Google Chrome but when switching platforms, in my case when running my code on Mac the interception works but when running the same code on Ubuntu the intercept callbacks never get called.
Let me know if it helps to provide more details, thanks.
git2samus
on 5 Mar 2019
@aslushnikov Happy March! Super exciting about the Firefox progress.
Just wondering what is an ETA on this issue? Thank you!
sradu
on 11 Mar 2019
@aslushnikov same here. When is this going to be fixed ? thanks in advanced
JOEAV
on 12 Mar 2019
Just in case it can help, another site that just won't load with request interception enabled, some URLs stays as "pending" and never resolve:
Fails with 1.13.0, 1.11.0
Works with 1.6.2
const puppeteer = require('puppeteer');
const URL = 'http://nowboarding.changiairport.com/';
(async () => {
const browser = await puppeteer.launch({
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
],
ignoreHTTPSErrors: true,
headless: false,
devtools: true,
});
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', request => request.continue());
const res = await page.goto(URL, {
timeout: 15000,
});
console.log(res.status());
await browser.close();
})();
Enabling NetworkService is not a fix for me. Using HeadlessChrome/75.0.3738.0, 302 redirect stays pending forever (in headless and non-headless modes)
I have tried using Puppeteer 1.13, next, and 1.14. Nothing works.
const browser = await puppeteer.launch(puppeteer_debug_opts);
const page = await browser.newPage();
let opts = {
method: 'POST',
url: `${get_test_url()}index.php`,
postData: mock_timer_object.create_post_data({
'timer_id': '24995',
'emp_id': '19',
'date': '2019-04-04',
'cust_id': '29',
'service_item': '3',
'class': '16',
'billable_default': '0',
'hours_ui': '5',
'minutes_ui': '8',
'duration': '308',
'tracker': '1231231',
'notes': 'bla+bla+bla',
'start_time': '0',
'start_pause_button': 'Resume'
}),
headers: {
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded'
}
};
await page.setRequestInterception(true);
page.once('request', request => {
console.log(request.url(), opts);
console.log("navigating to index");
console.log(request.postData()); // undefined
request.continue(opts);
console.log(request.postData()); // undefined
});
await page.goto(`${get_test_url()}index.php`);
await page.addScriptTag({path: require.resolve('jquery')});
browser.close();
kevinforrestconnors
on 4 Apr 2019
FWIW, for my integration testing solution, as an alternative to using puppeteer request interception, I ended up creating a local proxy server with Node.js using http-mitm-proxy and then recording/loading fixtures using nock. Nock intercepts requests by hijacking the Node.js core modules
httpandhttps, which the proxy uses.Requests are forced through the proxy server using the Chromum flags
--proxy-serverand--ignore-certificate-errors. The local Node.js proxy instead does https cert validation.
Would you mind providing more information about your fix? I can't figure out how nock plays into this.
kevinforrestconnors
on 4 Apr 2019
The fix has landed and will be available in the next release (will be published in the end of April / beginning of may 2019). 🎉
For now, please give it a try with 👉 npm i puppeteer@next
aslushnikov
on 9 Apr 2019
Hello @aslushnikov,
We are facing this issue with http internal website (not ssl supported), while accessing it with Puppeteer, Many thread related to this issue led me on this umbrella thread.
Below is the configurations tested on:
Puppeteer version: 1.12.2, 1.14.0, 1.14.0-next.1554871834316
Environment: Linux Containers (Using AWS ECS)
Puppeteer args used : [
'--no-sandbox',
'--disable-features=site-per-process',
'--window-size=500,500',
'--ignore-certificate-errors',
'--enable-features=NetworkService',
'--allow-running-insecure-content',
'--disable-web-security'
]
Other flags on page tried with :
await page.setExtraHTTPHeaders({
'upgrade-insecure-requests': '0'
});
await page.setRequestInterception(true)
page.on('request', interceptedRequest => {
interceptedRequest.continue()
})
Headless mode: true
In the error logs:
the request interception changes to https automatically although I have set 'upgrade-insecure-requests': '0' in headers.
Please note this issue arises only in linux containerized environment, Although this complete system works on a windows machine , in production and dev mode both for headless true and false.
Could you please align us to where exactly the cause could be as
Below is the error log:
Windows Envrionement where is this working perfectly: Puppeteer Version: 1.14.0, chromium version: Win64 - 641577 (local chromium)
{
"method": "Network.requestWillBeSent",
"params": {
"requestId": "8A328F784AA627A52ED4A050DFDE2C32",
"loaderId": "8A328F784AA627A52ED4A050DFDE2C32",
"documentURL": "http://****ACTUAL URL ****",
"request": {
"url": "http://****ACTUAL URL ****",
"method": "GET",
"headers": {
"Upgrade-Insecure-Requests": "0",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/75.0.3760.0 Safari/537.36"
},
"mixedContentType": "none",
"initialPriority": "VeryHigh",
"referrerPolicy": "no-referrer-when-downgrade"
},
"timestamp": 369.322781,
"wallTime": 1554896629.938341,
"initiator": {
"type": "other"
},
"type": "Document",
"frameId": "D1F3B248DD528CB7C9358BF92A36B5EF",
"hasUserGesture": false
},
"sessionId": "***SessionID****"
}
2019-04-10T11:43:50.158Z puppeteer:protocol ◀ RECV
{
"method": "Network.requestIntercepted",
"params": {
"interceptionId": "interception-job-1.2",
"request": {
"url": "https://**** REDIRECTED URL *****",
"method": "GET",
"headers": {
"Upgrade-Insecure-Requests": "0",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/75.0.3760.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
"Cookie": "ASP.NET_SessionId=vcrwcpfr2zqkxqiuspvzc255"
},
"initialPriority": "VeryHigh",
"referrerPolicy": "no-referrer-when-downgrade"
},
"frameId": "D1F3B248DD528CB7C9358BF92A36B5EF",
"resourceType": "Document",
"isNavigationRequest": true,
"requestId": "8A328F784AA627A52ED4A050DFDE2C32"
},
"sessionId": "***SessionID****"
}
md5-98ae49f02bb3fcbf9a78cd74d5291494
Error: at page.goto(URL)
{
"method": "Network.loadingFailed",
"params": {
"requestId": "8A328F784AA627A52ED4A050DFDE2C32",
"timestamp": 370.054342,
"type": "Document",
"errorText": "net::ERR_FAILED",
"canceled": false
},
"sessionId": "***SessionID****"
}
md5-dd2ef02ebb12d2ddb96abddfe44d4612
2019-04-10T11:43:49.318Z puppeteer:protocol SEND ► {"method":"Target.createBrowserContext","params":{},"id":2}
2019-04-10T11:43:49.319Z puppeteer:protocol ◀ RECV {"id":2,"result":{"browserContextId":"CB7F4584E9A57A656434E46C287AB2A5"}}
2019-04-10T11:43:49.320Z puppeteer:protocol SEND ► {"method":"Target.createTarget","params":{"url":"about:blank","browserContextId":"CB7F4584E9A57A656434E46C287AB2A5"},"id":3}
2019-04-10T11:43:49.328Z puppeteer:protocol ◀ RECV {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"D1F3B248DD528CB7C9358BF92A36B5EF","type":"page","title":"","url":"","attached":false,"browserContextId":"CB7F4584E9A57A656434E46C287AB2A5"}}}
2019-04-10T11:43:49.329Z puppeteer:protocol ◀ RECV {"id":3,"result":{"targetId":"D1F3B248DD528CB7C9358BF92A36B5EF"}}
2019-04-10T11:43:49.618Z puppeteer:protocol ◀ RECV {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"D1F3B248DD528CB7C9358BF92A36B5EF","type":"page","title":"about:blank","url":"about:blank","attached":false,"browserContextId":"CB7F4584E9A57A656434E46C287AB2A5"}}}
2019-04-10T11:43:49.619Z puppeteer:protocol SEND ► {"method":"Target.attachToTarget","params":{"targetId":"D1F3B248DD528CB7C9358BF92A36B5EF","flatten":true},"id":4}
2019-04-10T11:43:49.620Z puppeteer:protocol ◀ RECV {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"D1F3B248DD528CB7C9358BF92A36B5EF","type":"page","title":"about:blank","url":"about:blank","attached":true,"browserContextId":"CB7F4584E9A57A656434E46C287AB2A5"}}}
2019-04-10T11:43:49.620Z puppeteer:protocol ◀ RECV {"method":"Target.attachedToTarget","params":{"sessionId":"***SessionID****","targetInfo":{"targetId":"D1F3B248DD528CB7C9358BF92A36B5EF","type":"page","title":"about:blank","url":"about:blank","attached":true,"browserContextId":"CB7F4584E9A57A656434E46C287AB2A5"},"waitingForDebugger":false}}
2019-04-10T11:43:49.621Z puppeteer:protocol ◀ RECV {"id":4,"result":{"sessionId":"***SessionID****"}}
2019-04-10T11:43:49.623Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Page.enable","params":{},"id":5}
2019-04-10T11:43:49.624Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Page.getFrameTree","params":{},"id":6}
2019-04-10T11:43:49.625Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Target.setAutoAttach","params":{"autoAttach":true,"waitForDebuggerOnStart":false,"flatten":true},"id":7}
2019-04-10T11:43:49.625Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Performance.enable","params":{},"id":8}
2019-04-10T11:43:49.626Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Log.enable","params":{},"id":9}
2019-04-10T11:43:49.722Z puppeteer:protocol ◀ RECV {"id":5,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.723Z puppeteer:protocol ◀ RECV {"id":6,"result":{"frameTree":{"frame":{"id":"D1F3B248DD528CB7C9358BF92A36B5EF","loaderId":"9298ADF8B0AE00B772E9A92111C353C9","url":"about:blank","securityOrigin":"://","mimeType":"text/html"}}}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.724Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Page.setLifecycleEventsEnabled","params":{"enabled":true},"id":10}
2019-04-10T11:43:49.725Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Runtime.enable","params":{},"id":11}
2019-04-10T11:43:49.725Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Network.enable","params":{},"id":12}
2019-04-10T11:43:49.726Z puppeteer:protocol ◀ RECV {"id":7,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.734Z puppeteer:protocol ◀ RECV {"id":8,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.735Z puppeteer:protocol ◀ RECV {"id":9,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.741Z puppeteer:protocol ◀ RECV {"method":"Page.lifecycleEvent","params":{"frameId":"D1F3B248DD528CB7C9358BF92A36B5EF","loaderId":"9298ADF8B0AE00B772E9A92111C353C9","name":"commit","timestamp":368.924431}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.741Z puppeteer:protocol ◀ RECV {"method":"Page.lifecycleEvent","params":{"frameId":"D1F3B248DD528CB7C9358BF92A36B5EF","loaderId":"9298ADF8B0AE00B772E9A92111C353C9","name":"DOMContentLoaded","timestamp":368.924513}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.742Z puppeteer:protocol ◀ RECV {"method":"Page.lifecycleEvent","params":{"frameId":"D1F3B248DD528CB7C9358BF92A36B5EF","loaderId":"9298ADF8B0AE00B772E9A92111C353C9","name":"load","timestamp":369.000186}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.817Z puppeteer:protocol ◀ RECV {"method":"Page.lifecycleEvent","params":{"frameId":"D1F3B248DD528CB7C9358BF92A36B5EF","loaderId":"9298ADF8B0AE00B772E9A92111C353C9","name":"networkAlmostIdle","timestamp":369.000708}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.818Z puppeteer:protocol ◀ RECV {"method":"Page.lifecycleEvent","params":{"frameId":"D1F3B248DD528CB7C9358BF92A36B5EF","loaderId":"9298ADF8B0AE00B772E9A92111C353C9","name":"networkIdle","timestamp":369.000708}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.818Z puppeteer:protocol ◀ RECV {"id":10,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.818Z puppeteer:protocol ◀ RECV {"method":"Runtime.executionContextCreated","params":{"context":{"id":1,"origin":"://","name":"","auxData":{"isDefault":true,"type":"default","frameId":"D1F3B248DD528CB7C9358BF92A36B5EF"}}}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.819Z puppeteer:protocol ◀ RECV {"id":11,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.819Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Page.addScriptToEvaluateOnNewDocument","params":{"source":"//# sourceURL=__puppeteer_evaluation_script__","worldName":"__puppeteer_utility_world__"},"id":13}
2019-04-10T11:43:49.820Z puppeteer:protocol ◀ RECV {"id":12,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.821Z puppeteer:protocol ◀ RECV {"id":13,"result":{"identifier":"1"}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.821Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Page.createIsolatedWorld","params":{"frameId":"D1F3B248DD528CB7C9358BF92A36B5EF","grantUniveralAccess":true,"worldName":"__puppeteer_utility_world__"},"id":14}
2019-04-10T11:43:49.824Z puppeteer:protocol ◀ RECV {"method":"Runtime.executionContextCreated","params":{"context":{"id":2,"origin":"","name":"__puppeteer_utility_world__","auxData":{"isDefault":false,"type":"isolated","frameId":"D1F3B248DD528CB7C9358BF92A36B5EF"}}}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.824Z puppeteer:protocol ◀ RECV {"id":14,"result":{"executionContextId":2}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.825Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Emulation.setDeviceMetricsOverride","params":{"mobile":false,"width":800,"height":600,"deviceScaleFactor":1,"screenOrientation":{"angle":0,"type":"portraitPrimary"}},"id":15}
2019-04-10T11:43:49.825Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Emulation.setTouchEmulationEnabled","params":{"enabled":false},"id":16}
2019-04-10T11:43:49.917Z puppeteer:protocol ◀ RECV {"method":"Page.frameResized","params":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.920Z puppeteer:protocol ◀ RECV {"id":15,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.920Z puppeteer:protocol ◀ RECV {"id":16,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.921Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Network.setExtraHTTPHeaders","params":{"headers":{"upgrade-insecure-requests":"0"}},"id":17}
2019-04-10T11:43:49.922Z puppeteer:protocol ◀ RECV {"id":17,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.922Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Network.setCacheDisabled","params":{"cacheDisabled":true},"id":18}
2019-04-10T11:43:49.923Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Network.setRequestInterception","params":{"patterns":[{"urlPattern":"*"}]},"id":19}
2019-04-10T11:43:49.924Z puppeteer:protocol ◀ RECV {"id":19,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.925Z puppeteer:protocol ◀ RECV {"id":18,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:49.928Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Page.navigate","params":{"url":"http://****ACTUAL URL ****","frameId":"D1F3B248DD528CB7C9358BF92A36B5EF"},"id":20}
2019-04-10T11:43:49.940Z puppeteer:protocol ◀ RECV {"method":"Network.requestWillBeSent","params":{"requestId":"8A328F784AA627A52ED4A050DFDE2C32","loaderId":"8A328F784AA627A52ED4A050DFDE2C32","documentURL":"http://****ACTUAL URL ****","request":{"url":"http://****ACTUAL URL ****
2019-04-10T11:43:50.016Z puppeteer:protocol ◀ RECV {"method":"Network.requestIntercepted","params":{"interceptionId":"interception-job-1.0","request":{"url":"http://****ACTUAL URL ****","method":"GET","headers":{"Upgrade-Insecure-Requests":"0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Geck
2019-04-10T11:43:50.017Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Network.continueInterceptedRequest","params":{"interceptionId":"interception-job-1.0"},"id":21}
2019-04-10T11:43:50.020Z puppeteer:protocol ◀ RECV {"id":21,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:50.101Z puppeteer:protocol ◀ RECV {"method":"Network.requestWillBeSent","params":{"requestId":"8A328F784AA627A52ED4A050DFDE2C32","loaderId":"8A328F784AA627A52ED4A050DFDE2C32","documentURL":"http://**** REDIRECTED URL *****","r
2019-04-10T11:43:50.101Z puppeteer:protocol ◀ RECV {"method":"Network.requestIntercepted","params":{"interceptionId":"interception-job-1.1","request":{"url":"http://**** REDIRECTED URL *****","method":"GET","headers":{"Upgrade-Insecure-Request
2019-04-10T11:43:50.102Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Network.continueInterceptedRequest","params":{"interceptionId":"interception-job-1.1"},"id":22}
2019-04-10T11:43:50.103Z puppeteer:protocol ◀ RECV {"id":22,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:50.117Z puppeteer:protocol ◀ RECV {"method":"Network.requestIntercepted","params":{"interceptionId":"interception-job-1.1","request":{"url":"http://**** REDIRECTED URL *****","method":"GET","headers":{"Upgrade-Insecure-Request
2019-04-10T11:43:50.117Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Network.continueInterceptedRequest","params":{"interceptionId":"interception-job-1.1","authChallengeResponse":{"response":"ProvideCredentials","username":"AD-ONE\\jbuch","password":"Charlie1400!"}},"id":23}
2019-04-10T11:43:50.118Z puppeteer:protocol ◀ RECV {"id":23,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:50.157Z puppeteer:protocol ◀ RECV {"method":"Network.requestWillBeSent","params":{"requestId":"8A328F784AA627A52ED4A050DFDE2C32","loaderId":"8A328F784AA627A52ED4A050DFDE2C32","documentURL":"https://**** REDIRECTED URL *****","
2019-04-10T11:43:50.158Z puppeteer:protocol ◀ RECV {"method":"Network.requestIntercepted","params":{"interceptionId":"interception-job-1.2","request":{"url":"https://**** REDIRECTED URL *****","method":"GET","headers":{"Upgrade-Insecure-Reques
2019-04-10T11:43:50.158Z puppeteer:protocol SEND ► {"sessionId":"***SessionID****","method":"Network.continueInterceptedRequest","params":{"interceptionId":"interception-job-1.2"},"id":24}
2019-04-10T11:43:50.159Z puppeteer:protocol ◀ RECV {"id":24,"result":{}, "sessionId": "***SessionID****"}
2019-04-10T11:43:50.671Z puppeteer:protocol ◀ RECV {"method":"Network.loadingFailed","params":{"requestId":"8A328F784AA627A52ED4A050DFDE2C32","timestamp":370.054342,"type":"Document","errorText":"net::ERR_FAILED","canceled":false}, "sessionId": "***SessionID****"}
2019-04-10T11:43:50.674Z puppeteer:protocol ◀ RECV {"id":20,"result":{"frameId":"D1F3B248DD528CB7C9358BF92A36B5EF","loaderId":"8A328F784AA627A52ED4A050DFDE2C32","errorText":"net::ERR_FAILED"}, "sessionId": "***SessionID****"}
Error: net::ERR_FAILED at http://****ACTUAL URL ****
ngaba18
on 10 Apr 2019
For now, please give it a try with 👉
npm i puppeteer@next
I would love to try it but there is no puppeteer-core@next.
lo1tuma
on 10 Apr 2019
@ngaba18 from what I see in your logs, this seems to be a separate issue.
I'd love to look into it, but we'd need a repro. Any chance you can come up with one? Can you please file the issue separately?
I would love to try it but there is no puppeteer-core@next
@lo1tuma The majority of the work here happened upstream in chromium; chances are your locally installed chromium version doesn't have it yet.
aslushnikov
on 10 Apr 2019
@aslushnikov Good point, that’s probably true for the majority of users. For our project we are using a custom nix derivation to install chromium and we always try to use the same version as puppeteer does (in this case r649004).
lo1tuma
on 10 Apr 2019
The fix has landed and will be available in the next release (will be published in the end of April / beginning of may 2019).
For now, please give it a try with
npm i puppeteer@next
Sadly this didn't fix the issue for me. Using "puppeteer": "^1.14.0-next.1555713627875", chrome linux-650583 I still get hangs on 302s with setRequestInterception(true). Setting it to false and everything works like a charm.
ZachGoldberg
on 20 Apr 2019
@ZachGoldberg can you please file it separately with a repro? We'd be very happy to look into this!
aslushnikov
on 20 Apr 2019
The fix has landed and will be available in the next release (will be published in the end of April / beginning of may 2019).
For now, please give it a try withnpm i puppeteer@nextSadly this didn't fix the issue for me. Using
"puppeteer": "^1.14.0-next.1555713627875",chromelinux-650583I still get hangs on 302s with setRequestInterception(true). Setting it to false and everything works like a charm.
same here
JOEAV
on 21 Apr 2019
@JOEAV I'd appreciate if you can file a separate bug with a repro. We have a test that verifies that 302 redirects work, so something else should be going on.
aslushnikov
on 21 Apr 2019
The fix has landed and will be available in the next release (will be published in the end of April / beginning of may 2019).
For now, please give it a try with
npm i puppeteer@next
Fixed timeout navigation with request interception on, Thanks!
dembeleu
on 23 Apr 2019
I have the same issue on:
Puppeteer: 1.14.0-next.1556048038195
Chromium: r641577
OS: macOS 10.14.3 Mojave
await page.setRequestInterception(true);
page.on('request', req => {
req.continue();
});
Runs properly without it. What other information can I provide to debug this further?
ps-mariotacke
on 25 Apr 2019
@ps-mariotacke can you please file a separate issue with a script that reproduces it for you? We'll need to run it locally to debug what's going on.
aslushnikov
on 25 Apr 2019
Still can't work
Puppeteer: 1.15.0-next.1556265138389
Aaron-Bird
on 26 Apr 2019
if it's helpful to anyone else, i was able to avoid this issue by using the chromium version shipped with puppeteer. (no more executablePath arg)
cheers
spencermountain
on 26 Apr 2019
@Aaron-Bird if your scenario is different from the one in #4337, please share your repro in a separate issue.
aslushnikov
on 27 Apr 2019
I've been using Puppeteer to send fake responses to a web in my tests with the request interceptor. Up until [email protected] everything is working fine, however after updating to [email protected], fake requests with status code 422 are not being responded (400, 500 and 200 are working fine). Could that have something to do with the changes in this issue?.
await this._browser.page.setRequestInterception(true);
this._browser.page.on('request', request => {
request.respond({
status: 422, // 400 working fine
body: "foo"
});
});
I've only seen this happening with status 422
angrykoala
on 8 May 2019
@ngaba18 were you able to get your setup working? I may have run into the same issue with #4487.
nathagey
on 29 May 2019
@nathagey , My setup is still not working on Linux Container, though it works on windows machine.
ngaba18
on 30 May 2019
I facing similar issue some of the interception requests doesn't show in the callback for i.e. When I don't set request interception I'm able to see xhr url for image. When interception is on instead of showing url it gives response data (image data).
Because of this issue I'm not able to identify the url for which I should abort the request.
arif-b-khan
on 22 Jun 2019
Issue is still relevant in 2020.
Puppeteer Version: v3.3.0 (As packaged in Apify)
Node Version: v12.18.2
var puppeteerOptions = {
headless: true,
stealth: true,
useChrome: true
};
var browser = await Apify.launchPuppeteer(puppeteerOptions);
var page = (await browser.pages())[0]
page.setDefaultTimeout(0);
await page.setRequestInterception(true);
page.on('request', async (request) => {
//
});
console.log("going");
await page.goto("ANY HTTPS URL", {waitUntil: "domcontentloaded"});
console.log("gone");
"going" will log but "gone" never will.
WasabiThumb
on 10 Jul 2020
Moving page.setRequestInterception and page.on('request', ()=>{}) to AFTER page.goto works for me, but only because the request im scanning for happens for a determinate amount of time after the dom content loads. This won't work for everyone but it's worth a shot to see if the requests you wish to intercept happen late enough for this to work.
WasabiThumb
on 10 Jul 2020
@WasabiThumb Hey there, I had this issue. In my case, I was connecting to a remote chrome instance with a websocket url. The instance I was connecting to was created with an old version of puppeteer. I updated both that and my client (using puppeteer-core) to the latest (5.2.1 at the time of writing this comment) and it worked again.
Maybe updating to the latest or having matching versions will help you, too?
SamJarmanPP
on 23 Jul 2020
I can also confirm that updating to 5.2.1 fixed this issue for me.
ObviouslyGreen
on 15 Aug 2020
Related issues
sradu
·
45Comments
jakeg
·
47Comments
aherriot
·
172Comments
KJ1i
·
46Comments
optikalefx
·
49Comments
Most helpful comment
The fix has landed and will be available in the next release (will be published in the end of April / beginning of may 2019). 🎉
For now, please give it a try with 👉
npm i puppeteer@next