Chromeless: IOT endpoint Timed out after 30sec. Connection couldn't be established.

Created on 30 Jul 2017  Â·  50Comments  Â·  Source: prisma-archive/chromeless

Hi,

I keep running into the following issue when trying to call the proxy chrome set up when I try call chromeless with the remote set up.

"Timed out after 30sec. Connection couldn't be established."

I tried from Windows, Ubuntu and from another AWS Lambda always resulting in the above error message.

Have completed step 1 for the proxy set up and deployed The Proxy service to AWS Lambda.

Steps I have done

  • Changed the url for awsIotHost in the serverless.yml
  • Changed region in the serverless.yml
  • ran npm run deploy successfully and made note of the api key and url.

But the after that I am unable to call this proxy without running into the error "Timed out after 30sec. Connection couldn't be established.".

The script I am running is pretty simple i.e.

  const chromeless = new Chromeless({
    remote: {
      endpointUrl: 'https://xxxxxxxxxx.execute-api.eu-west-1.amazonaws.com/dev',
      apiKey: mykey
    }
  });

const links = await chromeless
    .goto('https://www.google.com')
    .type('chromeless', 'input[name="q"]')
    .press(13)
    .wait('#resultStats')
    .evaluate(() => {
      // this will be executed in headless chrome
      const links = [].map.call(document.querySelectorAll('.g h3 a'), a => ({
        title: a.innerText,
        href: a.href
      }));
      return JSON.stringify(links);
    })
    // you can still use the method chaining API after evaluating
    // when you're done, at any time you can call `.then` (in our case `await`)
    .scrollTo(0, 1000);

Always results in

node chrome.js
Error: Timed out after 30sec. Connection couldn't be established.
at Timeout._onTimeout (C:\dev\cb-headless-chrome\node_modules\chromeless\dist\src\chrome\remote.js:73:52)
at ontimeout (timers.js:488:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:283:5)

Is there some kind of permission, role I need that maybe is not in the documentation ?

Would love some help since I have no idea how to progress on this, even some idea which url it can't connect to ?

EDIT : After some more debugging I am finding its the IOT Host url that I can't connect to i.e.

wss://xxxxxxx.iot.eu-west-1.amazon.com/.....

Is there a step I am missing to allow connection to this ? I can see when I put my IOT endpoint url I can see the same kind of timeout behaviour.

Proxy help wanted question

Most helpful comment

Hi @danthegoodman @rfink @Kivol @furkanarius the Unable to import module 'src/run' and 'port' of undefined errors are now fixed in _master_ branch and will be available in the next release (v1.2) sometime later today/tomorrow. A temporary work-around is to downgrade the chromeless package to [email protected] in serverless/package.json.

All 50 comments

Hi @stevenfitzpatrick Hm.. To help debug this, let's confirm that the session Lambda handler is working correctly. Could you try the curl command I describe in https://github.com/graphcool/chromeless/issues/58#issuecomment-318603329 and share all of the result here?

Also, you've mentioned that you changed the region in serverless.yml. Is your region eu-west-1?

Yes region is eu-west-1

Output here

One thing I didn't understand from the documentation is which you consider as the "Session" GET end point.

I.e after npm run deploy, you get the following end points.

endpoints:
GET - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/version
OPTIONS - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/
GET - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/

Is that last get end point the one you consider the session one ?

@stevenfitzpatrick yes that's correct—GET /dev endpoint is the new-session endpoint. I'll clarify that in the documentation.

The only other thing I wanted to check was inserting the IOT Endpoint in the serverless.yml. I simply replaced the variable

"${env:AWS_IOT_HOST}"

with string

"xxxxxxxxxxx.iot.eu-west-1.amazonaws.com"

Is that also correct ?

The timeout seems to be happening here in chrome/remote.js

channel_1 = mqtt_1.connect(url, {
                          will: {
                            topic: 'chrome/last-will',
                            payload: JSON.stringify({ channelId: channelId_1 }),
                            qos: 1,
                            retain: false
                          }
                        });

Funny behaviour that I noticed when I print out the url it is connecting to.

wss://xxxxxxxxxx.iot.eu-west-1.amazon.com/mqtt?X-Amz-Date

The url has been changed from amazonaws.com to amazon.com at this point

@stevenfitzpatrick yes that's also correct (the AWS_IOT_HOST change you've made in serverless.yml) 😄

ok then I am kinda at a loss why it's not working. I am pretty sure it's due to this line

wss://xxxxxxxxx.eu-west-1.amazon.com

Is there maybe a role I need to add to my credentials, some kind of cert installed on my machine etc ?

Did the curl output help at all ?

@stevenfitzpatrick about amazonaws.com -> amazon.com .. that might be the problem. As far as I know it should be _amazonaws.com_ Something weird must be happening for the domain to get changed.. In aws4 package's code we use to create the presigned URL, there's no mention of amazon.com.

My chromeless version is 1.0.1 and serverless is 1.18.1

@adieuadieu Totally my fault I put down amazon.com instead of amazonaws.com in the serverless config.

my serverless version 1.8.1 , when use curl , i got `< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 815
< Connection: keep-alive
< Date: Wed, 02 Aug 2017 05:25:16 GMT
< x-amzn-RequestId: f7e6475d-7742-11e7-9732-d536e1f97d30
< X-Amzn-Trace-Id: sampled=0;root=1-5981623c-8fdf8fd359b663c2a046fa79
< X-Cache: Miss from cloudfront
< Via: 1.1 296daa1256a646119edb8b5abbf1c774.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: 6WVkXcnEiqM5h5pu-EYaBESjSrUmfcJfPjy5sAOoAEzPFE9HwZEl4Q==
<

but still got 30 seconds timeout , could anyone help me figure it out ? thanks

Just debug , the channel go to offline directly before it connected

@adieuadieu I'm having the same issue:
If I do the curl everything is fine:

*   Trying 54.230.11.103...
* TCP_NODELAY set
* Connected to xxxxxxx.execute-api.eu-west-2.amazonaws.com (54.230.11.103) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.eu-west-2.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> GET /dev/ HTTP/1.1
> Host: xxxxxxx.execute-api.eu-west-2.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> x-api-key: MY_API_KEY
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 831
< Connection: keep-alive
< Date: Thu, 03 Aug 2017 14:30:37 GMT
< x-amzn-RequestId: 517f1e75-7858-11e7-8b92-1976755a6dfc
< X-Amzn-Trace-Id: sampled=0;root=1-5983338d-3baf6c0a22e7da0a000369e8
< X-Cache: Miss from cloudfront
< Via: 1.1 2297ae65d06c070d63c49ed7259f5fbb.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: 8ioi901ViVi65L8YOH0pR9COW2YdxAmqeSUpTJoRTeJALmKvF_RmIQ==
<
* Curl_http_done: called premature == 0
* Connection #0 to host xxxxx.execute-api.eu-west-2.amazonaws.com left intact
{"url":"wss://https://xxxxx.iot.eu-west-2.amazonaws.com/mqtt?X-Amz-Date=20170803T143037Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASSAQEQIN6YSWTTMJH2A%2F20170803%2Feu-west-2%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=9b9fa351facf65c88364667acd84855a294d7d44cf08e7af4cf22b44621fa9dc&X-Amz-Security-Token=SOME_TOKEN","channelId":"cj5wjgoaf000201qcr7efo48q"}

Adding the debug: true I get:

WebSocket offline
Error: Timed out after 30sec. Connection couldn't be established.
    at Timeout._onTimeout (/Users/fabio/playground/chromeless-test/node_modules/chromeless/dist/src/chrome/remote.js:73:52)
    at ontimeout (timers.js:469:11)
    at tryOnTimeout (timers.js:304:5)
    at Timer.listOnTimeout (timers.js:264:5)

The test I'm trying to run:

const Chromeless = require('chromeless').default

async function run() {
  const chromeless = new Chromeless({
    remote: {
      endpointUrl: 'https://xxxxx.execute-api.eu-west-2.amazonaws.com/dev/',
      apiKey: 'xxxxxx'
    },
    debug: true
  })

  const screenshot = await chromeless
    .goto('https://www.google.com')
    .type('chromeless', 'input[name="q"]')
    .press(13)
    .wait('#resultStats')
    .screenshot()

  console.log(screenshot) // prints local file path or S3 url

  await chromeless.end()
}

run().catch(console.error.bind(console))

I double tested the values on the endpointUrl and apiKey with curl command and they work, so there's no wrong copy paste stuff

I decided to redo everything again, this time I using a different time region, eu-west-1

Still getting timeouts, but a different message when using debug:

Connected to message broker.
Error: Timed out after 30sec. Connection couldn't be established.
    at Timeout._onTimeout (/Users/fabio/playground/chromeless-test/node_modules/chromeless/dist/src/chrome/remote.js:73:52)
    at ontimeout (timers.js:469:11)
    at tryOnTimeout (timers.js:304:5)
    at Timer.listOnTimeout (timers.js:264:5)

the curl works fine fine:

*   Trying 54.230.11.142...
* TCP_NODELAY set
* Connected to xxxxxxx.execute-api.eu-west-1.amazonaws.com (54.230.11.142) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.eu-west-1.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> GET /dev HTTP/1.1
> Host: xxxxx.execute-api.eu-west-1.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> x-api-key: xxxxxx
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 825
< Connection: keep-alive
< Date: Thu, 03 Aug 2017 16:33:32 GMT
< x-amzn-RequestId: 7ce040e4-7869-11e7-a3e9-e326343e7894
< X-Amzn-Trace-Id: sampled=0;root=1-5983505c-2aa9848cd55bdb1bfaf046c8
< X-Cache: Miss from cloudfront
< Via: 1.1 7cddfedd82c4e2198668a64be4934fd0.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: jjwhqOOqqdiNzwvOvDPtn8XqlQSn1qgM4bCGVeDcz60T7A97QJd14A==
<
* Curl_http_done: called premature == 0
* Connection #0 to host xxxxxx.execute-api.eu-west-1.amazonaws.com left intact
{"url":"wss://xxxxxx.iot.eu-west-1.amazonaws.com/mqtt?X-Amz-Date=20170803T163332Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJUVCD2JRUATTLOUA%2F20170803%2Feu-west-1%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=3d1a5501384450b98cfde614ce023b3d8d304dc2fbae414e9d69845549ecffc4&X-Amz-Security-Token=token","channelId":"cj5wnuq9q000301qmplem18li"}

Hi @FabioAntunes at the moment I'm not really sure what might be causing this. I have a number of Proxy improvements and refactors I'm working on which will provide more verbose debug output for tracking these issues down.

We're getting the same error. Looking in the cloudwatch logs for the lambda run function, I'm seeing

Unable to import module 'src/run': Error

Follow up:
After adding in some console logs into the compiled code, it appears that there is an error when requiring the chromeless package.

It appears that this line is causing the lighthouse-logger to not be packaged, which is currently required by the chrome-launcher. After commenting it out on our end, we were able to get farther but something is still not right. Now we're seeing from the test suite:

TypeError: Cannot read property 'port' of undefined

There's no stacktrace attached to the error, so we're at a loss as to how to proceed.

Having an identical issue to @danthegoodman

I downloaded the chromeless zip from S3 and extracted and npm install again with node.js 6.10.3 and uploaded again, and I was able to pass the errors that @danthegoodman and @rfink faced, but now my CloudWatch log shows these messages and errors:

@serverless-chrome/lambda Spawning headless shell
@serverless-chrome/lambda ChromeLauncher No debugging port found on port 9222, launching a new Chrome.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: spawn EACCES

And in my local shell, I'm still getting 30sec the timeout error.

I'm having the same problem with @Kivol, and still found no solution. In fact, I followed the exact the same steps, but no good.

Hi @danthegoodman @rfink @Kivol @furkanarius the Unable to import module 'src/run' and 'port' of undefined errors are now fixed in _master_ branch and will be available in the next release (v1.2) sometime later today/tomorrow. A temporary work-around is to downgrade the chromeless package to [email protected] in serverless/package.json.

Thanks @adieuadieu ! Downgrading to @1.0.1 worked for @danthegoodman and I.

@adieuadieu I am still getting 'port' of undefined and I downgraded to chromeless 1.0.1. Is the next release still on track for release soon?

v1.2.0 has been released. Please update and let us know how it goes. Remember to do a npm install/yarn install after fetching/pulling latest from the repository.

I think I found one relevant bit here... Seems that the regions have to match currently. The iot endpoint has to be in the same region as the functions.

I've updated to the latest version this morning but am still seeing the same Unable to import module 'src/run': Error issue.

I also tried going back to 1.0.1 as suggested as a workaround but am getting the same result.

Is there anything I could be doing wrong? Really not sure what else to try

I've upgraded to v1.2.0 and tried removing the whole stack with ./node_modules/.bin/serverless and deployed again with npm run deploy.

I still see the following error from chromeless-serverless-dev-run lambda function log:

@serverless-chrome/lambda Spawning headless shell
@serverless-chrome/lambda ChromeLauncher No debugging port found on port 9222, launching a new Chrome.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: spawn EACCES

I've resolved the error above. Since I deployed in Windows, the Chrome binary seems to lack the +x permission when uploaded to S3 and lambda. When I tried deploying in Linux it works fine now!

Thanks for the support.

The Proxy won't work right when deployed from Windows 😞 . There's a known issue with Serverless packaging executables on Windows upstream in https://github.com/adieuadieu/serverless-chrome/issues/21 and upstream from there in https://github.com/serverless/serverless/issues/3557.

Two work-around solutions are in progress in serverless-chrome:

  • download chrome on lambda invocation (ick)
  • check chrome executable permission on invocation. If missing, copy chrome to /tmp (only place we have chmod permission), and chmod +x, then launch (ugh)

Both are lame, but it doesn't seem like the root issue will be addressed in https://github.com/serverless/serverless/issues/3557 anytime soon.

I can confirm all issues resolved from me, using Linux and chromeless 1.2.0. Great job guys! This project is moving quickly.

I have the same issue as @scosbet has...

Unable to import module 'src/run': Error

Did everything that suggest here, nothing helps.

Still having the same issue, getting the time out and Unable to import module 'src/run': Error in Lambda logs. Running off of master with Node 8.4 and built/deployed from OSX 10.12

hi @FabioAntunes , have you fixed the issue ? i still got this problem after upgrade to 1.3.0

@shawnLiujianwei I haven't used this in a while I'm waiting for Chromeless team make the switch to the Puppeteer.

What I did back then was pulling the master branch, remove all the lambda functions:

./node_modules/.bin/serverless remove

And then deploy them again following the instructions

@FabioAntunes got it , thanks

Anyone else running into this?
Timing out according to the function's max duration. At a loss.

Best way to debug this?

I've also been having trouble with this, I think others above are right, something's happening with the build/deploy.

EDIT: I spoke too soon, while I was talking to chromeless remotely I got hit by #414, when I followed by: Unable to import module 'src/run' which was resolved by building with node v6.10.0, however I'm now back to empty responses :(

I was deploying with node v6.10.3 but the problems went away when building with node v8.10.0 and I can still call it with node v6.10.3

There's a discrepancy on the deployed zip size:

  • node v6.10.0 -> 53.1M
  • node v6.10.3 -> 53.7M
  • node v8.10.0 -> 73.8M

I've also noticed that the serverless version:

  • deploys chromeless 1.3.0 (current version is 1.5.1)
  • the bucket naming in the serverless.yml doesn't take into account the stage
  • for now updating the engine required in the package.json to the earliest one that works may be enough

I'm working to a couple of deadlines at the moment otherwise I would drop in a PR

I experience the timeout error as well: "Chromeless Proxy disconnected due to inactivity (no commands sent for 30 seconds)." I've tested with the example provided in the readme.md.

I've done the same as OP: Changed the URL for awsIotHost and region in the serverless.yml and then deployed with npm run deploy which returns the API URL.

My setup is Ubuntu 16.04.4 LTS. I've deployed using Node JS 8.10 without any luck to AWS Lambda running Node JS 6.10. I've also tried with a fresh image of Ubuntu 16.04.4 LTS and Node JS 6.10, but no luck here.

I have the same issue with "chromeless": "1.5.1" and OSX. And with curl I have response 200

Chiming in with the same issue. It started consistently occurring once we had > 10 image tests.

Unfortunately I haven't been able to track down why the proxy is causing this issue. That said, it's becoming a very serious issue for our team as we're trying to to run 100s of tests but our CI can't consistently pass.

I'm happy to work on a PR if someone can help with some direction.

I do have the same issue. Anybody with more info?

Same issue here. Can curl with no problem and timeout occurs when running code. Shame, this was a perfect solution to something I was working on.
UPDATE: Added a slash to end of my endpoint, redeployed, and everything is working!

I'm also experiencing this issue - is there any guidance on where we should begin to look to potentially find a fix?

I am also having this issue. Is there any update on this?

I am also having this problem. Running on Node v8.1.0, Serverless 1.27.3, chromeless 1.5.2.

Curl works without any problems.

I get this output to the console...

Connected to message broker.
Error: Timed out after 30sec. Connection couldn't be established.
    at Timeout._onTimeout (/Users/.../node_modules/chromeless/dist/src/chrome/remote.js:73:52)
    at ontimeout (timers.js:488:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:283:5)

from line 96 of remote.ts....

channel.on('connect',` () => {
          if (this.options.debug) {
             console.log('Connected to message broker.')
          }

So.. from the output.. it looks like it gets here, but then nothing else happens so the timeout never gets cleared.

Any thoughts on what might be the cause of this? Or how I could debug this further?

Thanks!

@atebyagrue can you tell me more specific?

@cannalee90, after deploying the solution, the endpoints will be displayed. When I added one of these endpoints to my solution, I did add the trailing slash / at the end of the url. So, instead of
export CHROMELESS_ENDPOINT_URL=https://XYZXYZXYZ.execute-api.us-east-1.amazonaws.com/dev/
I used
export CHROMELESS_ENDPOINT_URL=https://XYZXYZXYZ.execute-api.us-east-1.amazonaws.com/dev
This is what was causing my failure, even though curl worked.

I have the same issue:

Connected to message broker.
Error: Timed out after 30sec. Connection couldn't be established.
    at Timeout._onTimeout (/Users/marekvybiral/projects/chromeless/serverless/node_modules/chromeless/dist/src/chrome/remote.js:73:52)
    at ontimeout (timers.js:475:11)
    at tryOnTimeout (timers.js:310:5)
    at Timer.listOnTimeout (timers.js:270:5)

Tried every solution suggested in this thread without any luck. 😥 It looks like it has no problem connecting to a WebSocket and subscribes to the mqtt thread but then no message is being received so it throws this error.

EDIT:
So I have redeployed with custom.debug: "*" instead of false in serverless.yml and now I get this error:

Connected to message broker.
Running remotely: {"type":"goto","url":"http://site.sponspack.com/jx76OP"}
Error: connect ECONNREFUSED 127.0.0.1:9222
Chromeless Proxy disconnected due to inactivity (no commands sent for 30 seconds).

EDIT 2:
Here is some error from CloudWatch logs:

2018-10-03T14:31:47.521Z    e3e8889c-c718-11e8-b92f-e984f73889d5
{
    "errorMessage": "Unable to start Chrome. If you have the DEBUG env variable set,there will be more in the logs.",
    "errorType": "Error",
    "stackTrace": [
        "/var/task/node_modules/@serverless-chrome/lambda/dist/bundle.cjs.js:376:13",
        "throw (native)",
        "step (/var/task/node_modules/@serverless-chrome/lambda/dist/bundle.cjs.js:311:193)",
        "/var/task/node_modules/@serverless-chrome/lambda/dist/bundle.cjs.js:311:404",
        "process._tickDomainCallback (internal/process/next_tick.js:135:7)"
    ]
}

HI, I am getting the same error. Anyone who was able to solve it is available to help?

error { error: Error: Timed out after 30sec. Connection couldn't be established.
4q9qv
2018-10-29 16:16:28-03:00 at Timeout._onTimeout (/app/bundle/programs/server/npm/node_modules/chromeless/dist/src/chrome/remote.js:73:52)
4q9qv
2018-10-29 16:16:28-03:00 at ontimeout (timers.js:475:11)
4q9qv
2018-10-29 16:16:28-03:00 at tryOnTimeout (timers.js:310:5)
4q9qv
2018-10-29 16:16:28-03:00 at Timer.listOnTimeout (timers.js:270:5)
4q9qv
2018-10-29 16:16:28-03:00 => awaited here:
4q9qv
2018-10-29 16:16:28-03:00 at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
4q9qv
2018-10-29 16:16:28-03:00 at Promise.asyncApply (imports/api/news/server/news-methods.js:22:7)
4q9qv
2018-10-29 16:16:28-03:00 at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 }
4q9qv
2018-10-29 16:16:30-03:00error { error: Error: Timed out after 30sec. Connection couldn't be established.
4q9qv
2018-10-29 16:16:30-03:00

@stevenfitzpatrick Have you found a solution ?
@pebay

Was this page helpful?
0 / 5 - 0 ratings

Related issues

queicherius picture queicherius  Â·  3Comments

adieuadieu picture adieuadieu  Â·  3Comments

ylscsdb picture ylscsdb  Â·  6Comments

just-boris picture just-boris  Â·  5Comments

bitflip2 picture bitflip2  Â·  6Comments