Aws-cdk: Can't use CDK to deploy to govcloud

[sentient-kshaffer]

When running cdk deploy from my command line when my AWS_PROFILE is set to my govcloud credentials. I get an error: Need to perform AWS calls for account unknown-account, but no credentials found. Tried: default credentials.. When I switch to a profile that is for a non-gov region, then it works just fine.

I have Administrator permissions in both accounts right now.

[rix0rrr]

Can you post a trace captured with -v?

[sentient-kshaffer]

@rix0rrr Sorry for the delay... I have run the trace here:

bash $ cdk deploy -v CDK toolkit version: 0.19.0 (build 2625a05) Command line arguments: { _: [ 'deploy' ], trace: false, strict: false, 'ignore-errors': false, ignoreErrors: false, json: false, j: false, verbose: true, v: true, ec2creds: undefined, i: undefined, 'version-reporting': undefined, versionReporting: undefined, 'path-metadata': true, pathMetadata: true, version: false, help: false, h: false, 'role-arn': undefined, r: undefined, roleArn: undefined, '$0': 'cdk', app: undefined, context: undefined, plugin: undefined, rename: undefined, profile: undefined, proxy: undefined, 'toolkit-stack-name': undefined, STACKS: [] } Determining whether we're on an EC2 instance. Does not look like EC2 instance. cdk.json: { "app": "node -r dotenv/config cloudformation.js" } Setting "aws:cdk:toolkit:default-region" context to us-gov-west-1 Resolving default credentials Looking up default account ID from STS Unable to determine the default AWS account (did you configure "aws configure"?): { InvalidClientTokenId: The security token included in the request is invalid. at Request.extractError (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29) at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:683:14) at Request.transition (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12) at /usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:685:12) at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18) message: 'The security token included in the request is invalid.', code: 'InvalidClientTokenId', time: 2018-12-31T16:59:18.579Z, requestId: '696ec36c-0d1d-11e9-8b5d-799bf40a45a1', statusCode: 403, retryable: false, retryDelay: 88.73410655454008 } Setting "aws:cdk:toolkit:default-account" context to undefined context: { 'aws:cdk:toolkit:default-region': 'us-gov-west-1', 'aws:cdk:toolkit:default-account': undefined, 'aws:cdk:enable-path-metadata': true } outdir: /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdkNvZgxs outfile: /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdkNvZgxs/cdk.out { version: '0.19.0', stacks: [ { name: 'MyStack', environment: [Object], template: [Object], metadata: [Object] } ], runtime: { libraries: { dotenv: '6.1.0', myLibrary: '1.0.0', '@aws-cdk/cdk': '0.19.0', '@aws-cdk/cx-api': '0.19.0', '@aws-cdk/assets-docker': '0.19.0', '@aws-cdk/aws-cloudformation': '0.19.0', '@aws-cdk/aws-codepipeline-api': '0.19.0', '@aws-cdk/aws-events': '0.19.0', '@aws-cdk/aws-iam': '0.19.0', '@aws-cdk/aws-ecr': '0.19.0', '@aws-cdk/aws-lambda': '0.19.0', '@aws-cdk/aws-cloudwatch': '0.19.0', '@aws-cdk/aws-ec2': '0.19.0', '@aws-cdk/aws-s3-notifications': '0.19.0', '@aws-cdk/aws-sqs': '0.19.0', '@aws-cdk/aws-kms': '0.19.0', '@aws-cdk/assets': '0.19.0', '@aws-cdk/aws-s3': '0.19.0' } } } Removing outdir /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdkNvZgxs Stack name not specified, so defaulting to all available stacks: MyStack Need to perform AWS calls for account unknown-account, but no credentials found. Tried: default credentials. Error: Need to perform AWS calls for account unknown-account, but no credentials found. Tried: default credentials. at CredentialsCache.getCredentials (/usr/local/lib/node_modules/aws-cdk/lib/api/util/sdk.ts:191:11) at <anonymous>

[rix0rrr]

Hi @sentient-kshaffer, thanks for getting back to us.

Can you try again with version 0.20.0 or higher? It has this fix which I hope should fix this issue.

[randallprince]

I'm having a similar issue here (0.22.0 (build 644ebf5)). As a note, I can run stuff like Amplify without any issues. This fails with "Need to perform AWS calls for account unknown-account, but no credentials found. Tried: default credentials." error.

[rix0rrr]

Can you please run the command again with -v and paste the output?

[mikeder]

Having the same issue here, I've tried aws configure and setting the AWS env var's directly with no luck. Also, may be worth noting that my ~/.aws/credentials are for a "root" account and I generally set the AWS_PROFILE env var to switch roles from my ~/.aws/config

mike.eder@MEDER1-MBK:infra2$ cdk list -v
CDK toolkit version: 0.22.0 (build 644ebf5)
Command line arguments: { _: [ 'list' ],
  trace: false,
  strict: false,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  verbose: true,
  v: true,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  version: false,
  help: false,
  h: false,
  long: false,
  l: false,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  '$0': '/usr/local/bin/cdk',
  app: undefined,
  context: undefined,
  plugin: undefined,
  rename: undefined,
  profile: undefined,
  proxy: undefined,
  'toolkit-stack-name': undefined }
Determining whether we're on an EC2 instance.
Does not look like EC2 instance.
cdk.json: {
  "app": "node bin/infra2.js",
  "region": "us-east-1",
  "account": "223582410118"
}
Setting "aws:cdk:toolkit:default-region" context to us-east-1
Resolving default credentials
Unable to determine the default AWS account (did you configure "aws configure"?): TypeError: Cannot redefine property: default
    at Function.defineProperty (<anonymous>)
    at /usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/shared-ini/ini-loader.js:11:14
    at Array.forEach (<anonymous>)
    at IniLoader.parseFile (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/shared-ini/ini-loader.js:8:26)
    at IniLoader.loadFrom (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/shared-ini/ini-loader.js:56:30)
    at SharedIniFileCredentials.load (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/credentials/shared_ini_file_credentials.js:105:44)
    at SharedIniFileCredentials.coalesceRefresh (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/credentials.js:205:12)
    at SharedIniFileCredentials.refresh (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/credentials/shared_ini_file_credentials.js:190:10)
    at SharedIniFileCredentials.get (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/credentials.js:122:12)
    at resolveNext (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/credentials/credential_provider_chain.js:125:17)
Setting "aws:cdk:toolkit:default-account" context to undefined
context: { 'aws:cdk:toolkit:default-region': 'us-east-1',
  'aws:cdk:toolkit:default-account': undefined,
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true }
outdir: /var/folders/g0/tpl82vys559dhqjk6sn4mmtsw3_td0/T/cdkkLKNTF
outfile: /var/folders/g0/tpl82vys559dhqjk6sn4mmtsw3_td0/T/cdkkLKNTF/cdk.out
{ version: '0.19.0',
  stacks:
   [ { name: 'Infra2Stack',
       environment: [Object],
       missing: [Object],
       template: [Object],
       metadata: [Object] } ],
  runtime:
   { libraries:
      { '@aws-cdk/cdk': '0.22.0',
        '@aws-cdk/cx-api': '0.22.0',
        '@aws-cdk/aws-ec2': '0.22.0',
        '@aws-cdk/aws-s3': '0.22.0',
        '@aws-cdk/aws-iam': '0.22.0',
        '@aws-cdk/aws-kms': '0.22.0',
        '@aws-cdk/aws-s3-notifications': '0.22.0',
        '@aws-cdk/aws-codepipeline-api': '0.22.0',
        '@aws-cdk/aws-events': '0.22.0',
        'jsii-runtime': 'node.js/v11.7.0' } } }
Removing outdir /var/folders/g0/tpl82vys559dhqjk6sn4mmtsw3_td0/T/cdkkLKNTF
Some context information is missing. Fetching...
Reading AZs for 223582410118:us-east-1
Need to perform AWS calls for account 223582410118, but no credentials found. Tried: default credentials.
Error: Need to perform AWS calls for account 223582410118, but no credentials found. Tried: default credentials.
    at CredentialsCache.getCredentials (/usr/local/lib/node_modules/aws-cdk/lib/api/util/sdk.ts:191:11)

[RafalWilinski]

I had similar issue and it seems that the problem was corrupted ~/.aws/config file. After cleaning it, everything went well.

[mikeder]

Creating a new service user in the sub account I'm targeting with CDK and then using their credentials in the [default] block of my credentials file resolved this for me. It seems as though CDK doesn't respect my /.aws/config, which during normal aws cli usage properly assumes the configured role.

I'd expect to not have to generate service user credentials for each sub account in my config in order to use CDK.

[rix0rrr]

This is an interesting error: TypeError: Cannot redefine property: default.

From the stack trace I can tell this is happening somewhere in the AWS SDK for JavaScript during the loading of your ~/.aws/config file, but without being able to see it it will be hard to guess at what.

For future reference to other people in this thread, I've made a topic that clearly spells out where the CDK's authentication mechanisms are incompatible with the AWS CLI:

https://github.com/awslabs/aws-cdk/issues/1656

[rix0rrr]

@mikeder https://github.com/aws/aws-sdk-js/blob/master/lib/shared-ini/ini-loader.js#L11

To me this seems like it would happen if you have [default] or maybe [profile default] multiple times in your ~/.aws/config.

[mikeder]

Ah, @rix0rrr you're right. I just took a look at my original ~/.aws/config and had both a [default] and a [profile default] block. I removed the [profile default] entry and reverted my credentials file to my master account credentials and CDK seems to be working as expected.

Thanks for reference on the auth mechanisms too 👍

[tunagami]

I am having this problem WITHOUT a [profile default] entry. Any idea why I am getting the same error?

[default]
region = eu-central-1

[profile int-server]
source_profile = default
role_arn = arn:aws:iam::121212121212:role/deployer
mfa_serial = arn:aws:iam::421212121212:mfa/[email protected]

[profile prod-server]
source_profile = default
role_arn = arn:aws:iam::321212121212:role/deployer
mfa_serial = arn:aws:iam::421212121212:mfa/[email protected]

error:

Setting "aws:cdk:toolkit:default-region" context to eu-central-1
Resolving default credentials
Unable to determine the default AWS account (did you configure "aws configure"?): { AccessDenied: Access denied

[rix0rrr]

@tunagami have you looked at this issue: https://github.com/awslabs/aws-cdk/issues/1656 ?

I would imagine it has something to do with the mfa_serial.

[tunagami]

@tunagami have you looked at this issue: #1656 ?

I would imagine it has something to do with the mfa_serial.

I re-posted my issue in the issue. Thank you @rix0rrr