Aws-cdk: Toolkit: bail out early if credentials are not defined

[eladb]

Seems like currently even if there are no credentials defined in the default chain, the toolkit will still execute Looking up default account ID from STS which will time out and only then a message is displayed that credentials are not found.

We should be able to bail out early if there are no credentials defined in the chain (we should make sure that plugins can still do their job)

[RomainMuller]

@eladb how do we know there is no credentials? Is there a SDK/client way to test upfront? Because my understanding is the way to do is try to make a call and realize it fails... And if that takes long on the STS call, it's because it tries to contact EC2 Metadata Service, which can be client-disabled...

[rix0rrr]

Yeah the timeout happens in determining whether there are credentials available. Specifically, whether there are instance creds.

Once there aren't, the actual call to STS is not even attempted. (But the logging can't tell so it looks misleading)

[rix0rrr]

We can configure a credential chain w/o instance creds, but then it won't work on EC2/CodeBuild. Add a switch?

[RomainMuller]

On machines that are not EC2, one can add this to their shell profile to disable EC2 metadata service lookups "globally" (at least for the JS SDK):

export AWS_EC2_METADATA_DISABLED=1

[rix0rrr]

Yeah, sure, but that's not the default. Just wondering out loud if it makes sense to invert the default in our case

[RomainMuller]

I disagree. People may well use an EC2 instance as their primary environment & rely on the instance profile. The behavior I want to be aiming for is "same as SDK / CLI", so I would like to deviate form the defaults as little as possible.

[eladb]

I am actually realizing that in order to obtain the account ID we are issuing the STS call in every toolkit invocation, which is terribly sad. Maybe we can cache it somehow...

For example, can we instantiate the default credentials provider in the toolkit (before the STS cal) and request credentials.

• If there are no credentials - bail out
• if there are credentials, use the access key as a key into an on-disk cache from key to account ID. If there is no entry in the cache, fetch account ID from STS.

[RomainMuller]

If doing the disk cache, consider that a lot of people are going to be dependent on temporary credentials (EC2 instance profile, federated credentials, ...). Don't want that file to grow forever.

[eladb]

Don't want that file to grow forever.
Good point

[rix0rrr]

People may well use an EC2 instance as their primary environment & rely on the instance profile

They may, but I honestly don't expect this to be a common case.

Anyhoo, on-disk cache of size 1 fixes the slowness, so we can totally do that.

[RomainMuller]

Cloud9?

[rix0rrr]

Can't imagine they rely on instance profile creds. But maybe they do.

[rix0rrr]

This might be useful for an optimization:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/identify_ec2_instances.html

On Lambda we get environment variables prepopulated.

Don't know about ECS.

[0xdevalias]

I've just run into this issue myself (from #702), I tend not to have default creds set, but use a profile (which I hadn't specified), takes forever to eventually get the timeout from 169.254.169.254 being unreachable:

Resolving default credentials
Unable to determine the default AWS account (did you configure "aws configure"?): { Error: connect ETIMEDOUT 169.254.169.254:80
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1113:14)
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT',
  syscall: 'connect',
  address: '169.254.169.254',
  port: 80 }
Setting "aws:cdk:toolkit:default-account" context to undefined
node bin/poc-aws-cdk.js 'base64:eyJ0eXBlIjoibGlzdCIsImNvbnRleHQiOnsiYXdzOmNkazp0b29sa2l0OmRlZmF1bHQtcmVnaW9uIjoiYXAtc291dGhlYXN0LTIifX0='

A couple of questions:

• As mentioned above, could we detect in a better way when in a 'cloud like' environment, and only then attempt to use the 'magic IP'
• Could we turn down the timeout on the connect request to something smaller (eg. 5sec), which would presumably be more than enough to hit an 'internal address' when running on a cloud-like environment
• How does a command such as aws sts get-caller-identity handle this sort of situation? As for me, it figures out instantly that I don't have creds defined:

⇒  aws sts get-caller-identity
Unable to locate credentials. You can configure credentials by running "aws configure".