Aws-sdk-js: Request hangs on missing credentials

[medikoo]

• [x] I've gone though Developer Guide and API reference
• [x] I've checked AWS Forums and StackOverflow for answers

---

When I attempt to run request without an AWS credentials it just hangs infinitely.

I'd expect it to crash immediately with Missing credentials error.

Why it's not the case?

e.g. following will just hang if run without any AWS creds setup:

const { Lambda } = require('aws-sdk');
const lambda = new Lambda();

lambda
  .invoke({ FunctionName: 'foo' })
  .promise()
  .then(
    data => console.log('Success', data),
    error => console.log('Error', data)
  );

[leonaves]

Encountered this error with a typo in my access credential properties, cost a lot of time trying to work out the issue because the SDK never reported an error.

[sziraqui]

Same error. Noticed when m Dockerfile did not have .aws/credentials present. No error reported, the SecretsManager.getSecrets call just waits forever.

[sziraqui]

Solved the problem:

import { promisify } from 'util';
function async getInstance() {
    const credChain = new CredentialProviderChain();
    try {
      promisify(credChain.resolve); // credChain.resolvePromise() does not catch errors
      const creds = await promisify(credChain.resolve)();
      return new SecretsManager({ region: 'your_region', credentials: creds });
    }
    catch (error) {
      throw // your custom error
    }
}

[ajredniwja]

@medikoo can you please check if you have any typos inn your credentials file?

[medikoo]

@ajredniwja why it was closed?

@medikoo can you please check if you have any typos inn your credentials file?

Have you read the issue description, it's about error that happens when there's no credentials file. Instead proper error being reported we observe a process lock and no information given.

Can you please reopen

[ajredniwja]

@medikoo It was closed due to lack of response, but I believe I mixed up two different issues(apologies).

While it should not be the case that the request just hangs for infinite time as I was not able to reproduce the behavior, there might be something else triggering that.

Can you please share your use and the SDK version?

[medikoo]

I was not able to reproduce the behavior, there might be something else triggering that.

It happens whenever we try to invoke SDK in any non-AWS enviroment, e.g. on your local computer.

I've already confirmed with AWS support, that it happens because AWS (when not finding credentials locally) tries to communicate with EC2 Metadata Service, which obviously cannot be accessed in non AWS network environments.

Issue is that AWS doesn't fail immediately when EC2 Metadata Service is inaccessible in given network environment, but instead it issues a long network request which eventually times out after 1 minute or more.

[ajredniwja]

I tried it by removing the creds from the default location.

Issue is that AWS doesn't fail immediately when EC2 Metadata Service is inaccessible in given network environment, but instead it issues a long network request which eventually times out after 1 minute or more.

Yes that might be the case, but since it times out, it should still throw the error after sometime(1 minute).

[medikoo]

Yes that might be the case, but since it times out, it should still throw the error after sometime(1 minute).

I see processing being on stall, with no feedback for a user for over 1 minute a bug.
Most users don't know what's going on and do "Ctrl-C", with no hint given it's unlikely to assume that credentials are missing

[pauloprestes]

aws-sdk-go checks if the timeout is set to 0 for the ec2metada and overrides it to 1 second, to solve this issue, maybe we could do the same here?
https://github.com/aws/aws-sdk-go/blob/0c96ad536ae45eb23a2b5ed2b4ef9a840eec6f6e/aws/ec2metadata/service.go#L76

on aws-sdk-js, it is always set to 0.
https://github.com/aws/aws-sdk-js/blob/2d1be139f5d95b4846e5c99a474f08a7c093c440/lib/metadata_service.js#L38

[ajredniwja]

@medikoo I understand the concern there and it can be frustrating at times. I believe this is not necessarily a bug but the behavior of the SDK which can be enhanced. Hence would like to mark it as a feature request.

Also you can use the workaround mentioned in the comment at
https://github.com/aws/aws-sdk-js/blob/2d1be139f5d95b4846e5c99a474f08a7c093c440/lib/metadata_service.js#L32

Using the timeout param as mentioned here.

The team would welcome a PR if you'd like to work on it.

[medikoo]

Thanks @ajredniwja I'll try to look into that

[pauloprestes]

Hey @ajredniwja I've seen this behavior happening on lambdas with high volume of invocations, like 1 on 1k invocations, maybe even less.
Generally it is easy solved by adding a timeout and retries.

[ajredniwja]

@pauloprestes I agree with you

Issue is that AWS doesn't fail immediately when EC2 Metadata Service is inaccessible in given network environment, but instead it issues a long network request which eventually times out after 1 minute or more.

As a workaround you can introduce your custom timeout as the inaccessible metadata takes the longest.

[patrobinson]

How can you overwrite the timeout?
I tried:

  aws.MetadataService.httpOptions = { timeout: 100 };

But it's still taking ~2.5min to fail

[pauloprestes]

How can you overwrite the timeout?
I tried:

  aws.MetadataService.httpOptions = { timeout: 100 };

But it's still taking ~2.5min to fail

We do this:

const s3 = new S3({
  apiVersion: '2006-03-01',
  signatureVersion: 'v4',
  httpOptions: {
    connectionTimeout: 1000,
    timeout: 1000
  },
  region: 'ap-southeast-2',
  maxRetries: 3
})

This changes do not work to fix this issue.

[pauloprestes]

@pauloprestes I agree with you

Issue is that AWS doesn't fail immediately when EC2 Metadata Service is inaccessible in given network environment, but instead it issues a long network request which eventually times out after 1 minute or more.

As a workaround you can introduce your custom timeout as the inaccessible metadata takes the longest.

@ajredniwja, Any reason to not set a default timeout to MetadataService? As I pointed before, other aws sdk does that.

[patrobinson]

@pauloprestes I tried this on node v13.11 and SDK versions v2.655.0 and v2.656.0 but it still hangs

Code to replicate

/tmp $ time node index.js
Error: connect ETIMEDOUT 169.254.169.254:80
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)

real    2m32.504s
user    0m0.245s
sys 0m0.041s

[pauloprestes]

Hey @patrobinson thanks for your replay.
I've done some tests using the code that you provide, and you are correct, that code doesn't fix this issue.
However, I've created a small PR to fix this issue on the SDK, it turns out it is a simple fix. I couldn't find any solution without changing the SDK.

[leonaves]

@ajredniwja

Regarding your statement:

I understand the concern there and it can be frustrating at times. I believe this is not necessarily a bug but the behavior of the SDK which can be enhanced. Hence would like to mark it as a feature request.

I'm not sure if anyone has mentioned/encountered this, but when I run the SDK with invalid credentials a few times, it occasionally does error with MissingCredentials, but usually hangs. This definitely would classify as a bug to me, if the behaviour is inconsistent. It should either hang consistently or throw an error consistently.

[patrobinson]

Agree with @leonaves. I can't replicate this at present but colleagues have it happen consistently in their environments.

[medikoo]

Workaround seems to run following before aws-sdk is required: (Appeared as not safe, see: https://github.com/serverless/serverless/issues/7997)

if (!process.env.AWS_EC2_METADATA_DISABLED) {
  const nslookupResult = require('child_process').spawnSync('nslookup', ['169.254.169.254']);
  if (
    nslookupResult.status === 1 ||
    // Windows case
    (nslookupResult.status === 0 &&
      String(nslookupResult.stderr).includes("can't find 169.254.169.254"))
  ) {
    process.env.AWS_EC2_METADATA_DISABLED = '1';
  }
}

const AWS = require('aws-sdk');

[medikoo]

I've diagnosed it further and it appears that are exactly two issues in a code:

1. httpOptions.timeout as defined here: https://github.com/aws/aws-sdk-js/blob/e229c41d420759b08fb08e4723fbcc652cf4080d/lib/metadata_service.js#L31-L38 is ineffective, due to: https://github.com/aws/aws-sdk-js/blob/e229c41d420759b08fb08e4723fbcc652cf4080d/lib/http/node.js#L43 (I believe this line should be removed, also statement by it is not true)

2. By default there's no timeout for MetadataService query, I believe there should be some sane default set.

Due to (1), the only reliably working workaround I see at this point, is to patch Node.js http.request as follows:

const http = require('http');
const originalRequest = http.request;
http.request = function(url, options, cb) {
  if (url && url.host === '169.254.169.254' && !('timeout' in url)) {
    // Workaround https://github.com/aws/aws-sdk-js/issues/2983
    url.timeout = 1000;
  }
  return originalRequest.call(this, url, options, cb);
};

[mhart]

@medikoo have you tried connectTimeout? That worked for me last time I checked this issue

EDIT: connectTimeout, not connectionTimeout

[mhart]

Here's an example of how I used it in cmda: https://github.com/lambci/cmda/blob/cb599e5c11d6702ee9c349fab81ab4eb667eb284/cli/aws.js#L99

[medikoo]

@mhart great thanks for pointing that. It indeed works. We can enforce aws-sdk to follow same by default, via patch as follows:

const metadataServicePrototype = require('aws-sdk/lib/metadata_service').prototype;

const originalRequest = metadataServicePrototype.request;

metadataServicePrototype.request = function(path, options, callback) {
  this.maxRetries = 0;
  if (!this.httpOptions.connectTimeout) this.httpOptions.connectTimeout = 1000;
  return originalRequest.call(this, path, options, callback);
};

[iteratelance]

Spent countless hours trying to figure this out. Upgrading to aws-sdk 2.737.0 seems to have solved the issue for me.

[ajredniwja]

Closing in favor of https://github.com/aws/aws-sdk-js/pull/3356