Serverless-chrome: Issue with a freeze between Serverless-chrome and pyppeteer on AWS Lambda

[alessbelli]

I am trying to setup an automated browsing pipeline for a python AWS Lambda, and I'm very happy to have landed on this project, it's awesome so far! (locally)

I use pyppeteer (puppeteer port for python), with the standalone serverless-chrome binary you provided. Everything works as expected both locally AND on a 'lambda environment simulator' on docker, but NOT on AWS.

I set the user data directory and the executable paths so that everything stays inside the only writable directory for a lambda, /tmp.

There is no explicit error, the lambda simply timeouts after any duration I give it.
--> After the browser launches, I try to open a page, and then it freezes.
I set logging level to DEBUG and enabled a 'dumpio' option on pyppeteer that allows to see websocket calls between the browser and the driver, and this gives me the following:

on AWS

[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:35827/devtools/browser/0c43dfe3-ec23-4465-8dbb-2b87b1295593
[D:pyppeteer.connection] SEND: {"id": 1, "method": "Target.setDiscoverTargets", "params": {"discover": true}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"3DF1768EA540672AA08ABDD9062F0888","type":"page","title":"","url":"about:blank","attached":false,"browserContextId":"9CD7E80AC2FDAFB30A5145A06826A388"}}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"479d2580-8fac-48ee-a490-2f4f62742bcd","type":"browser","title":"","url":"","attached":true}}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"6b9f15c4-2c66-47d6-86b0-1a42f4503479","type":"browser","title":"","url":"","attached":false}}}
[D:pyppeteer.connection] RECV: {"id":1,"result":{}}
launched browser
[D:pyppeteer.connection] SEND: {"id": 2, "method": "Target.createTarget", "params": {"url": "about:blank"}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"58B8AE6B20FBBE129E5FB238C8D04155","type":"page","title":"","url":"","attached":false,"browserContextId":"9CD7E80AC2FDAFB30A5145A06826A388"}}}
[D:pyppeteer.connection] RECV: {"id":2,"result":{"targetId":"58B8AE6B20FBBE129E5FB238C8D04155"}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"58B8AE6B20FBBE129E5FB238C8D04155","type":"page","title":"","url":"about:blank","attached":false,"browserContextId":"9CD7E80AC2FDAFB30A5145A06826A388"}}}
[D:pyppeteer.connection] SEND: {"id": 3, "method": "Target.attachToTarget", "params": {"targetId": "58B8AE6B20FBBE129E5FB238C8D04155"}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"58B8AE6B20FBBE129E5FB238C8D04155","type":"page","title":"","url":"about:blank","attached":true,"browserContextId":"9CD7E80AC2FDAFB30A5145A06826A388"}}}
[D:pyppeteer.connection] RECV: {"method":"Target.attachedToTarget","params":{"sessionId":"58B8AE6B20FBBE129E5FB238C8D04155:1","targetInfo":{"targetId":"58B8AE6B20FBBE129E5FB238C8D04155","type":"page","title":"","url":"about:blank","attached":true,"browserContextId":"9CD7E80AC2FDAFB30A5145A06826A388"},"waitingForDebugger":false}}
[D:pyppeteer.connection] RECV: {"id":3,"result":{"sessionId":"58B8AE6B20FBBE129E5FB238C8D04155:1"}}
[D:pyppeteer.connection] SEND: {"id": 4, "method": "Target.sendMessageToTarget", "params": {"sessionId": "58B8AE6B20FBBE129E5FB238C8D04155:1", "message": "{\"id\": 1, \"method\": \"Page.enable\", \"params\": null}"}}
[D:pyppeteer.connection] RECV: {"id":4,"result":{}}
END RequestId: c505dbaa-a6ef-11e8-ba68-8581b737a228
REPORT RequestId: c505dbaa-a6ef-11e8-ba68-8581b737a228  Duration: 300100.13 ms  Billed Duration: 300000 ms  Memory Size: 1024 MB    Max Memory Used: 167 MB 
2018-08-23T16:20:40.247Z c505dbaa-a6ef-11e8-ba68-8581b737a228 Task timed out after 300.10 seconds

on Docker-lambda

[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:33989/devtools/browser/4db9d517-3ff3-4a89-98a1-9b3dcc4d7302
[D:pyppeteer.connection] SEND: {"id": 1, "method": "Target.setDiscoverTargets", "params": {"discover": true}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"560c6416-82ae-4b74-8e06-1ba619568e03","type":"browser","title":"","url":"","attached":true}}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"866be95f-9cc5-4e0f-ba1c-0ce5db40cd65","type":"browser","title":"","url":"","attached":false}}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"C17947E0EEC25F1C55896445646DD00E","type":"page","title":"","url":"about:blank","attached":false,"browserContextId":"ABA11A894ADF1B6E780954685D38F886"}}}
[D:pyppeteer.connection] RECV: {"id":1,"result":{}}
launched browser
[D:pyppeteer.connection] SEND: {"id": 2, "method": "Target.createTarget", "params": {"url": "about:blank"}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"AEF87E850DF475433350EE7DDF03266C","type":"page","title":"","url":"","attached":false,"browserContextId":"ABA11A894ADF1B6E780954685D38F886"}}}
[D:pyppeteer.connection] RECV: {"id":2,"result":{"targetId":"AEF87E850DF475433350EE7DDF03266C"}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"C17947E0EEC25F1C55896445646DD00E","type":"page","title":"about:blank","url":"about:blank","attached":false,"browserContextId":"ABA11A894ADF1B6E780954685D38F886"}}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"AEF87E850DF475433350EE7DDF03266C","type":"page","title":"about:blank","url":"about:blank","attached":false,"browserContextId":"ABA11A894ADF1B6E780954685D38F886"}}}
[D:pyppeteer.connection] SEND: {"id": 3, "method": "Target.attachToTarget", "params": {"targetId": "AEF87E850DF475433350EE7DDF03266C"}}
[D:pyppeteer.connection] RECV: {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"AEF87E850DF475433350EE7DDF03266C","type":"page","title":"about:blank","url":"about:blank","attached":true,"browserContextId":"ABA11A894ADF1B6E780954685D38F886"}}}
[D:pyppeteer.connection] RECV: {"method":"Target.attachedToTarget","params":{"sessionId":"AEF87E850DF475433350EE7DDF03266C:1","targetInfo":{"targetId":"AEF87E850DF475433350EE7DDF03266C","type":"page","title":"about:blank","url":"about:blank","attached":true,"browserContextId":"ABA11A894ADF1B6E780954685D38F886"},"waitingForDebugger":false}}
[D:pyppeteer.connection] RECV: {"id":3,"result":{"sessionId":"AEF87E850DF475433350EE7DDF03266C:1"}}
[D:pyppeteer.connection] SEND: {"id": 4, "method": "Target.sendMessageToTarget", "params": {"sessionId": "AEF87E850DF475433350EE7DDF03266C:1", "message": "{\"id\": 1, \"method\": \"Page.enable\", \"params\": null}"}}
[D:pyppeteer.connection] RECV: {"id":4,"result":{}}
[D:pyppeteer.connection] RECV: {"method":"Target.receivedMessageFromTarget","params":{"sessionId":"AEF87E850DF475433350EE7DDF03266C:1","message":"{\"id\":1,\"result\":{}}","targetId":"AEF87E850DF475433350EE7DDF03266C"}}
[D:pyppeteer.connection] SEND: {"id": 5, "method": "Target.sendMessageToTarget", "params": {"sessionId": "AEF87E850DF475433350EE7DDF03266C:1", "message": "{\"id\": 2, \"method\": \"Page.getFrameTree\", \"params\": null}"}}
[D:pyppeteer.connection] RECV: {"id":5,"result":{}}
[D:pyppeteer.connection] RECV: {"method":"Target.receivedMessageFromTarget","params":{"sessionId":"AEF87E850DF475433350EE7DDF03266C:1","message":"{\"id\":2,\"result\":{\"frameTree\":{\"frame\":{\"id\":\"AEF87E850DF475433350EE7DDF03266C\",\"loaderId\":\"1CDDD3D5E90C4AD55EA7C433B1C82A9D\",\"url\":\"about:blank\",\"securityOrigin\":\"://\",\"mimeType\":\"text/html\"}}}}","targetId":"AEF87E850DF475433350EE7DDF03266C"}}

of course in the second case the navigation continues for a while so this is only what I thought was the relevant part.
Also, possibly unrelated, when I enable this level of debug, I also get Fontconfig warnings for mltiple in in both cases but since it does not affect the outcome in my local environment I doubt that.

It looks as if the chromium binary does not respond to the instruction to open a page, and therefore the driver is stuck in that position.

Any insights?
Will also submit the issue to pyppeteer since this issue involves possibly one or the other library.

[adieuadieu]

With which flags are you starting Chromium with? Try it with the following:

--disable-dev-shm-usage --disable-gpu --single-process --no-zygote --no-sandbox

[alessbelli]

already had --disable-gpu and --no-sandbox, probably the --single-process that did it since the new page likeley corresponds to a new process that (maybe?) cannot spawn inside a lambda, but that fixed it!!!

Thank you _a million times_, will refer to this to the devs at docker-lambda since their docker image fails to replicate this behavior!

[DinisCruz]

Thanks for the info above, that really helped. If it helps others, here is how I was able to do it : https://twitter.com/DinisCruz/status/1063023374920769542