Chromeless: AWS Lambda error timeout after 30s / "cannot execute binary file"

[tnolet]

Hi,

just did a fresh install of Chromeless with the remote option. The lambda function is being triggered when I run the example script, but it times out after 30 secs. I followed the instructions exactly and the deployment seemed to have gone perfectly. I can see that Chrome is simply not starting, probably caused by the fact the the "headless_shell" binary cannot be run.

This is 100% reproducable, I could give your my api key to try it out.

Data:

chromeless 1.3.0
aws region: eu-central-1

Cloudwatch output:

TART RequestId: 29c90ac5-8e3d-11e7-944e-07f9e1b67b80 Version: $LATEST
2017-08-31T11:11:43.192Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Spawning headless shell
2017-08-31T11:11:43.198Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda ChromeLauncher No debugging port found on port 9222, launching a new Chrome.
2017-08-31T11:11:43.206Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Launcher Chrome running with pid 13 on port 9222.
2017-08-31T11:11:43.207Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 0
2017-08-31T11:11:43.711Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 1
2017-08-31T11:11:44.211Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 2
2017-08-31T11:11:44.712Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 3
2017-08-31T11:11:45.213Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 4
2017-08-31T11:11:45.714Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 5
2017-08-31T11:11:46.216Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 6
2017-08-31T11:11:46.717Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 7
2017-08-31T11:11:47.218Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 8
2017-08-31T11:11:47.719Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 9
2017-08-31T11:11:48.220Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Waiting for Chrome 10
2017-08-31T11:11:48.230Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda Error trying to spawn chrome: { Error: connect ECONNREFUSED 127.0.0.1:9222
at Object.exports._errnoException (util.js:1018:11)
at exports._exceptionWithHostPort (util.js:1041:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 9222 }
2017-08-31T11:11:48.236Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda stdout log: 
2017-08-31T11:11:48.236Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    @serverless-chrome/lambda stderr log: /var/task/node_modules/@serverless-chrome/lambda/dist/headless_shell: /var/task/node_modules/@serverless-chrome/lambda/dist/headless_shell: cannot execute binary file

2017-08-31T11:11:48.236Z    29c90ac5-8e3d-11e7-944e-07f9e1b67b80    Error occured in serverless-plugin-chrome wrapper when trying to ensure Chrome for default() handler.
{}
Error: Unable to start Chrome. If you have the DEBUG env variable set,there will be more in the logs.
at /var/task/node_modules/@serverless-chrome/lambda/dist/bundle.cjs.js:304:13
at throw (native)
at step (/var/task/node_modules/@serverless-chrome/lambda/dist/bundle.cjs.js:257:191)
at /var/task/node_modules/@serverless-chrome/lambda/dist/bundle.cjs.js:257:402
at process._tickDomainCallback (internal/process/next_tick.js:135:7)
END RequestId: 29c90ac5-8e3d-11e7-944e-07f9e1b67b80
REPORT RequestId: 29c90ac5-8e3d-11e7-944e-07f9e1b67b80  Duration: 5048.54 ms    Billed Duration: 5100 ms Memory Size: 1536 MB   Max Memory Used: 38 MB  

[adieuadieu]

@tnolet Are you deploying from Windows? Deploying the Proxy from Windows currently won't work because of a known issue: https://github.com/graphcool/chromeless/issues/70#issuecomment-318634457, https://github.com/adieuadieu/serverless-chrome/issues/21, https://github.com/serverless/serverless/issues/3557

[tnolet]

@adieuadieu No, I'm deploying from MacOS. I am however running Node 6.11.2. Is it possible that the installation scripts need Node 8? I'm aware the Chromeless syntax uses Node 8 features, and that is fine.

[kamilregulski]

@tnolet @adieuadieu I have the same error: Timed out after 30sec. Connection couldn't be established. (MacOS, Node: 8.4.0, chromeless 1.3.0, eu-central-1)

[tnolet]

@kamilregulski can you maybe check your Cloudwatch logs and see if it has the same cannot execute binary file error?

[kamilregulski]

@tnolet Yes, the same:

2017-09-01T22:50:20.373Z eb74e766-8f67-11e7-93ce-531cc778d996 @serverless-chrome/lambda stderr log: /var/task/node_modules/@serverless-chrome/lambda/dist/headless_shell: /var/task/node_modules/@serverless-chrome/lambda/dist/headless_shell: cannot execute binary file

[jorgeer]

Have the exact same issue.

[tnolet]

Maybe something changed in the AWS lambda container, making the binary incompatibe?

[carlosbaraza]

What steps could we follow to debug this?

Should we rebuild the binary?

[carlosbaraza]

After thinking about it for a while, I think I found the root of the issue. The AMI for the Lambdas was updated to the 2017.03 version.

I ran a simple lambda doing cat /etc/system-release and this was the output:

"Amazon Linux AMI release 2017.03\n"

Tomorrow I will try to recompile Chrome 61 with the Lambda patch using the "Amazon Linux AMI release 2017.03" image.

[carlosbaraza]

I found a workaround, though it is a bit annoying. https://github.com/adieuadieu/serverless-chrome/issues/65

At least, I know the root cause, and we can work on a solution.

[carlosbaraza]

There was a bug in Serverless CLI 1.21.0... https://github.com/serverless/serverless/pull/4189

We can close this ticket.

Solution

Update Serverless to 1.21.1.

npm install -g [email protected]

[tnolet]

@carlosbaraza fantastic! Thanks for the hard work!

[sanchitbh]

Just want to add that if you are getting this error then please make sure your AWS_IOT region is same as the region specified in serverless.yml (it defaults to ireland i think). I was getting this timeout error because of this and now it's fixed.

P.S. Also since it's my first comment here, a huge thanks to everyone involved in this project for their hardwork. This is just awesome!

[Raidus]

hmm..none of the mentioned solutions are working in my case.

configuration:
serverless: 1.21.1
chromeless: 1.3.0
operating system: Mac
node: 8.2.1

in my serverless.yml

awsIotHost: ${env:XXXXXXXXXXX.iot.eu-central-1.amazonaws.com}
runtime: nodejs6.10
region: eu-central-1

Note: I cannot see any logs in Cloudwatch in /aws/lambda/chromeless-serverless-dev-run

During deploying I'm getting follow warnings:

 Serverless Warning --------------------------------------

  A valid environment variable to satisfy the declaration 'env:a2aq5tbva8bu8c.iot.eu-central-1.amazonaws.com' could not be found.


 Serverless Warning --------------------------------------

  A valid environment variable to satisfy the declaration 'env:a2aq5tbva8bu8c.iot.eu-central-1.amazonaws.com' could not be found.


 Serverless Warning --------------------------------------

  A valid service attribute to satisfy the declaration 'self:custom.awsIotHost' could not be found.

/tests/chromeless/chromeless/serverless/node_modules/@types/graphql/subscription/subscribe.d.ts (17,4): Cannot find name 'AsyncIterator'.
/tests/chromeless/chromeless/serverless/node_modules/@types/graphql/subscription/subscribe.d.ts (29,4): Cannot find name 'AsyncIterable'.

any ideas?