Aws-sdk-js: SQS Inaccessible host

[hixus]

For some reason sqs fails time to time with error message:
Inaccessible host: sqs.eu-west-1.amazonaws.com'. This service may not be available in theeu-west-1' region.

This is weird because the script starts fine and gets messages correctly but after 1 or 2 days it throws the mentioned message.

[AdityaManohar]

This error is thrown when the SDK encounters a lower level ENOTFOUND error. This is usually because of a transient network or DNS issue.

Would you be able to verify network / DNS connectivity of the environment running your code? I'd be interested to know what you find.

Hope this helps!

[mick]

I have got this error to on a few requests to DynamoDB, from an several EC2s in the same region.

UnknownEndpoint: Inaccessible host: `dynamodb.us-east-1.amazonaws.com'. This service may not be available in the `us-east-1' region.
    at Request.ENOTFOUND_ERROR (/usr/local/src/app/node_modules/aws-sdk/lib/event_listeners.js:358:46)
    at Request.callListeners (/usr/local/src/app/node_modules/aws-sdk/lib/sequential_executor.js:100:18)
    at Request.emit (/usr/local/src/app/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/local/src/app/node_modules/aws-sdk/lib/request.js:604:14)
    at ClientRequest.error (/usr/local/src/app/node_modules/aws-sdk/lib/event_listeners.js:204:22)
    at ClientRequest.<anonymous> (/usr/local/src/app/node_modules/aws-sdk/lib/http/node.js:62:19)
    at ClientRequest.emit (events.js:117:20)
    at CleartextStream.socketErrorListener (http.js:1551:9)
    at CleartextStream.emit (events.js:95:17)
    at Socket.onerror (tls.js:1455:17)

Since this seems caused by intermittent DNS request failures, shouldnt this error be retryable:true? https://github.com/aws/aws-sdk-js/blob/master/lib/event_listeners.js#L362

cc @lsegal

[lsegal]

That's a fair point regarding retryability. The primary intention of this error is to deal with clients that are incorrectly configured to talk to services in regions that they are not available on (as displayed in the message). In those cases, retrying would be wasteful. We could probably toggle retryable on for this class of errors without a significant behavioral difference for those users if it means being able to potentially recover from intermittent networking issues.

[lsegal]

FWIW you should be able to use the following global event in the meantime to make this class of errors retryable:

AWS.events.on('httpError', function() {
  if (this.response.error && this.response.error.code === 'UnknownEndpoint') {
    this.response.error.retryable = true;
  }
});

I haven't tested this myself, so let me know if it does not work.

[mick]

@lsegal great thanks. I'll try that out. In the meantime I sent a PR for the change #529

[lsegal]

@mick thanks, we'll do some testing to make sure there's no unwanted impact but otherwise I don't see a problem with taking that change.

[AdityaManohar]

I did some testing with this. It doesn't seem like there is any noticeable impact for clients talking to incorrectly configured regions.

This also means that it is unlikely that the retry backoff is long enough to catch transient network issues that are of a longer duration.

[devinus]

@AdityaManohar This is actually super useful for us. Right now I suspect we're having issues with DNS resolve caching in Docker containers, so at high load dns.resolve just fails to work and we get these errors (maybe 1 in 50 times). Obviously the "correct" solution is to figure out why this is happening in the first place, but for now this is going to make it much more robust for us. Thanks @mick !

[vladgolubev]

In case anyone is still here. For me it appears when I miss http://in the queue url.

[GeoffreyPlitt]

Getting this error a lot, none of the workarounds above work for me.

[rsmolkin]

I've also gotten this error a few times, and one time, I left a process running, and it just resumed after a few fails on its own.

[kodayashi]

I was doing development work at the office, then went home and restarted my node app and started getting this. I ran sudo killall -HUP mDNSResponder on my Mac (macOS) to flush the DNS cache and boom the errors went away.

[Matho]

Hi @hixus , @GeoffreyPlitt . Did you solve this problem? If yes, could you please share the solution with us? I have similar problem in #2618 Thanks

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.