Terragrunt: AWS CLIv2 and AWS SSO auth fails

[stevie-]

We are using new aws cli v2 SSO auth feature. This works with aws cli, but terragrunt fails to get credentials.

$ aws --version
aws-cli/2.0.6 Python/3.8.2 Darwin/18.7.0 botocore/2.0.0dev10

.aws/config

[profile 123456789012-AWSAdministratorAccess]
sso_start_url = https://d-123456789.awsapps.com/start
sso_region = eu-west-1
sso_account_id = 123456789012
sso_role_name = AWSAdministratorAccess
region = eu-west-1
output = json

Check aws cli works

$ export AWS_PROFILE=123456789012-AWSAdministratorAccess
$ aws sso login
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://device.sso.eu-west-1.amazonaws.com/

Then enter the code:

REDACTED
Successully logged into Start URL: https://d-123456789.awsapps.com/start
$ aws s3 ls
2020-04-08 08:09:44 tf-states.lab.REDACTED

$ aws sts get-caller-identity
{
    "UserId": "REDACTED:redacted@redacted",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/AWSReservedSSO_AWSAdministratorAccess_abcdef12345/redacted@redacted"
}
$ terragrunt plan --terragrunt-source ~/REDACTED/git/tf-modules//aws/bootstrap
[terragrunt] 2020/04/09 10:38:44 Terragrunt Version: v0.23.6
[terragrunt] 2020/04/09 10:38:44 Reading Terragrunt config file at /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/terragrunt.hcl
[terragrunt] 2020/04/09 10:38:44 Did not find any locals block: skipping evaluation.
[terragrunt] 2020/04/09 10:38:44 Running command: /Users/REDACTED/git/cloud-config/terraform/../scripts/aws/get_admin_roles.sh
AWSReservedSSO_AWSAdministratorAccess_abcdef12345[terragrunt] 2020/04/09 10:38:46 run_cmd output: [AWSReservedSSO_AWSAdministratorAccess_abcdef12345]
[terragrunt] 2020/04/09 10:38:46 Found locals block: evaluating the expressions.
[terragrunt] 2020/04/09 10:38:46 Evaluated 5 locals (remaining 8): cloud, environment, default_empty_yaml, api_domain, aws_vars
[terragrunt] 2020/04/09 10:38:46 Evaluated 3 locals (remaining 5): terraform_module_local_path, module_settings_path, aws_root_domain
[terragrunt] 2020/04/09 10:38:46 Evaluated 2 locals (remaining 3): terraform_module_path, terraform_module_version
[terragrunt] 2020/04/09 10:38:46 Evaluated 3 locals (remaining 0): terraform_module_name, default_tags, terraform_source_default
[terragrunt] [/Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap] 2020/04/09 10:38:51 Running command: terraform --version
[terragrunt] 2020/04/09 10:38:51 Downloading Terraform configurations from file:///Users/REDACTED/git/tf-modules into /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro
[terragrunt] 2020/04/09 10:38:51 Copying files from /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap into /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap
[terragrunt] 2020/04/09 10:38:51 Setting working directory to /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap
[terragrunt] 2020/04/09 10:38:51 The file path /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap/provider.tf already exists and if_exists for code generation set to "overwrite". Regenerating file.
[terragrunt] 2020/04/09 10:38:51 Generated file /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap/provider.tf.
[terragrunt] 2020/04/09 10:38:51 The file path /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap/backend.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file.
[terragrunt] 2020/04/09 10:38:51 Generated file /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap/backend.tf.
[terragrunt] [/Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap] 2020/04/09 10:38:51 Initializing remote state for the s3 backend
[terragrunt] 2020/04/09 10:38:51 Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[terragrunt] 2020/04/09 10:38:51 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[stevie-]

S3 bucket config from terragrunt.hcl. We checked setting AWS profile here as well. No change.

# Configure Terragrunt+Terraform to automatically store tfstate files in an S3 bucket
remote_state {
  backend = "s3"

  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }

  config = {
    bucket         = "tf-states.${local.environment}.${local.aws_root_domain}"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "eu-central-1"
    encrypt        = true
    dynamodb_table = "tf-states-lock.${local.environment}.${local.aws_root_domain}"

    s3_bucket_tags      = local.default_tags
    dynamodb_table_tags = local.default_tags
    # save costs - since we can not add lifecycle rules to clean up old versions
    skip_bucket_versioning = true
  }
}

[brikis98]

I believe the new SSO functionality only works with the newest version of the AWS SDK, so all CLI tools that use the SDK (such as Terragrunt) would need to be updated first. Would you be up for a PR to bump the AWS SDK version? You could test locally to see if that fixes the issue for you.

[stevie-]

PR = PullRequest -- my GOlang is very basic. I doubt that I can handle it.

Found this changelog:
https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md#release-v12530-2019-11-07

v1.25.30 added support for SSO, maybe related.

[stevie-]

I checked the following, but no success.

11:05 $ git diff
diff --git a/go.mod b/go.mod
index c958777..4f9ec28 100644
--- a/go.mod
+++ b/go.mod
@@ -4,7 +4,7 @@ go 1.13

 require (
        cloud.google.com/go/storage v1.6.0
-       github.com/aws/aws-sdk-go v1.29.34
+       github.com/aws/aws-sdk-go v1.30.7
        github.com/creack/pty v1.1.9
        github.com/fatih/color v1.9.0
        github.com/fatih/structs v1.1.0
diff --git a/go.sum b/go.sum
index 1042b8b..7426f99 100644
--- a/go.sum
+++ b/go.sum
@@ -99,6 +99,8 @@ github.com/aws/aws-sdk-go v1.25.3/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN
 github.com/aws/aws-sdk-go v1.27.1/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
 github.com/aws/aws-sdk-go v1.29.34 h1:yrzwfDaZFe9oT4AmQeNNunSQA7c0m2chz0B43+bJ1ok=
 github.com/aws/aws-sdk-go v1.29.34/go.mod h1:1KvfttTE3SPKMpo8g2c6jL3ZKfXtFvKscTgahTma5Xg=
+github.com/aws/aws-sdk-go v1.30.7 h1:IaXfqtioP6p9SFAnNfsqdNczbR5UNbYqvcZUSsCAdTY=
+github.com/aws/aws-sdk-go v1.30.7/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
 github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f/go.mod h1:AuiFmCCPBSrqvVMvuqFuk0qogytodnVFVSN5CeJB8Gc=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
 github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
@@ -330,6 +332,8 @@ github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANyt
 github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
 github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM=
 github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
+github.com/jmespath/go-jmespath v0.3.0 h1:OS12ieG61fsCg5+qLJ+SsW9NicxNkg3b25OyT2yCeUc=
+github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik=
 github.com/joefitzgerald/rainbow-reporter v0.1.0/go.mod h1:481CNgqmVHQZzdIbN52CupLJyoVwB10FQ/IQlF1pdL8=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/joyent/triton-go v0.0.0-20180313100802-d8f9c0314926/go.mod h1:U+RSyWxWd04xTqnuOQxnai7XGS2PrPY2cfGoDKtMHjA=

Build new version with go build and used this.
But same Error:

[terragrunt] [/Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap] 2020/04/09 11:04:37 Initializing remote state for the s3 backend
[terragrunt] 2020/04/09 11:04:37 Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[terragrunt] 2020/04/09 11:04:37 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[lorengordon]

AWS SSO is going to be a mess across the community because it does not make credentials available in a way that works with the current credential chain. Each SDK will need to implement support for the new credential chain, and perhaps each tool also (depending on how they setup their chain).

For GO, see: https://github.com/aws/aws-sdk-go/issues/3186

[jsaevitzon]

I also started getting these errors last night around 7pm.

[brikis98]

Ah, good catch @lorengordon... If this isn't implemented in the AWS Go SDK, then we won't be able to do anything until it is.

[lorengordon]

Fwiw, here is a workaround using credential_process until more SDKs support the AWS SSO credential chain...

https://github.com/benkehoe/aws-sso-credential-process