Terragrunt: Error finding AWS credentials for S3 remote state

[VladimirShushkov]

Hi guys,

Kindly asking for your advice. I see that that the issue was previously discussed several times from different angles. I can't make our TG code working with my creds in AWS multiaccount environments without creating dedicated IAM user in all these AWS accounts. I am successfully using assume-role consumption and aws cli is perfectly working so I can pull data from all accounts just by setting appropriate AWS_PROFILE and AWS_DEFAULT_REGION where profiles are configured in ~/.aws/config this way

[profile main]
output = json
region = eu-central-1

[profile dev]
output = json
region = eu-central-1
role_arn = arn:aws:iam::xxxxxxxxxx:role/AdminRole
source_profile = main

What I need to do is to create resources in AWS dev account where we have s3 remote state stored as well using IAM user from main account via assume-role approach.

Terragrunt code I am trying to push looks like this, for example, which has variables specifications only and is pulling TF module config from git repo

terragrunt = {
  terraform {
    source = "git::path"
  }

  # Include all settings from the root terraform.tfvars file
  include = {
    path = "${find_in_parent_folders()}"
  }
}

solution_owner                = "Devops"
vpc_cidr_block                = "10.0.1.0/20"
...

It also takes TG configs from parent folders where I have remote state config

terragrunt = {
  # Configure Terragrunt to automatically store tfstate files in an S3 bucket
  remote_state {
    backend = "s3"
    config {
      encrypt        = true
      bucket         = "dev-terraform-state"
      key            = "${path_relative_to_include()}/terraform.tfstate"
      region         = "eu-central-1"
      dynamodb_table = "terraform-locks"
      profile        = "dev"
    }
  }
  # Configure root level variables that all resources can inherit
  terraform {
    extra_arguments "bucket" {
      commands = ["${get_terraform_commands_that_need_vars()}"]
      optional_var_files = [
          "${get_tfvars_dir()}/${find_in_parent_folders("account.tfvars", "ignore")}"
      ]
    }
  }
}

During execution of terragrunt plan I am getting the error pointing to absent creds for reaching remote state s3

[terragrunt] [...] 2019/03/06 12:01:40 Running command: terraform --version
[terragrunt] 2019/03/06 12:01:40 Reading Terragrunt config file at .../core/vpc/terraform.tfvars
[terragrunt] 2019/03/06 12:01:40 WARNING: no double-slash (//) found in source URL /tf/module-aws-vpc.git. Relative paths in downloaded Terraform code may not work.
[terragrunt] 2019/03/06 12:01:40 Cleaning up existing *.tf files in .../core/vpc/.terragrunt-cache/VSGljja7WSjrKw1hwRMSENKXBAA/yAyfAK-S9z7ucSeeqfftLQhM-MA
[terragrunt] 2019/03/06 12:01:40 Downloading Terraform configurations from git::ssh://asdasd.git?ref=pr/12 into .../core/vpc/.terragrunt-cache/VSGljja7WSjrKw1hwRMSENKXBAA/yAyfAK-S9z7ucSeeqfftLQhM-MA using terraform init
[terragrunt] [.../core/vpc] 2019/03/06 12:01:40 Initializing remote state for the s3 backend
[terragrunt] 2019/03/06 12:02:00 Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[terragrunt] 2019/03/06 12:02:00 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

I tried to use all 3 approaches described in section "Work with multiple AWS accounts" of main TG repo page with setting AWS_PROFILE env. variable, using sts assume-role etc, but I am stuck on the same stage of init remote s3.

As I see TG supported assume-role approach for quite long time. Is there any caveats or limitations with that? What am I doing wrong?

Please help.

Thank you very much! #

[brikis98]

  remote_state {
    backend = "s3"
    config {
      encrypt        = true
      bucket         = "dev-terraform-state"
      key            = "${path_relative_to_include()}/terraform.tfstate"
      region         = "eu-central-1"
      dynamodb_table = "terraform-locks"
      profile        = "dev"
    }
  }

Is the profile = "dev" intentional?

[jvanwagner]

I'm getting a similar issue myself except I get this error:

Error configuring the backend "s3": No valid credential sources found for AWS Provider.
Please see https://terraform.io/docs/providers/aws/index.html for more information on
providing credentials for the AWS Provider

Please update the configuration in your Terraform files to fix this error
then run this command again.

[JoshiiSinfield]

@VladimirShushkov @jvanwagner have you seen #616 ? It sounds like you could be experiencing similar.

Cheers,
Josh

[rafaeloening-barigui]

+1

[brianpham]

Any updates on this? I am seeing the same issue on my end

[khdevel]

I think I figured it out... I will test it deeply but for now it works! My scenario is as follow:

• I have my IAM User in the _Management_ AWS Account
• this user has a Role which allows to Assume the IAM Role on _Dev_ AWS Account
• the IAM Role at _Dev_ account is available after the MFA process

So I think that is something like above problem plus MFA which increases the complexity... but the solution is quite simple - I had to have a [default] profile in my ~/.aws/credentials which seems to be mandatory for Terragrunt! Of course other profiles are valid but this one is something like a _must-have_

My ~/.aws/credentials file was as follow:

[default]
aws_access_key_id = AKIA...
aws_secret_access_key = ry5tgFree...

[assumeHelper]
role_arn = arn:aws:iam::123456789012:role/FooRole
mfa_serial = arn:aws:iam::098765432109:mfa/foo.bar
region = eu-central-1
source_profile = default

[mfaAssume]
aws_access_key_id = ASIA...
aws_secret_access_key = wi47gPZR...
aws_session_token = FwoGZXIv...

[default] - contains the data for the IAM User at _Management_ AWS Account
[assumeHelper] - it is a "notepad" for the aws sts command, not used in any Terraform's configuration
[mfaAssume]- contains the data which was generated by the aws sts command

Then my terragrunt.hcl looked like below which the most important variable is profile = "mfaAssume"

remote_state {
  backend = "s3"
  config = {
    bucket         = "foo-terraform-state"
    dynamodb_table = "foo-terraform-state"
    encrypt        = true
    key            = "terraform.tfstate"
    profile        = "mfaAssume"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "eu-central-1"
  }
}

This configuration worked properly! I did not have any messages from Terragrunt like

Error finding AWS credentials in file '~/.aws/credentials' (did you set the correct file name and/or profile?): NoCredentialProviders: no valid providers in chain. Deprecated

or

Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated.

TEST

To be sure that it is all about the default profile I testes it by:

1. renaming [default] to some other e.g. [myProfile] - DID NOT WORK
2. leave the [default] as an empty profile and put its previous credentials to other e.g. [myProfile] - DID NOT WORK

During the tests my variable profile had the same value mfaAssume.

CONCLUSION
It seems that for Terragrunt somehow the profile [default] must be present in case we want to use some other. I do not know whether it is because of some relation between [mfaAssume] and [default] or some other reason...

[brikis98]

Thx for doing the research @khdevel. Under the hood, Terragrunt (and Terraform) both use the AWS Go SDK. Perhaps we're somehow using it wrong, but I'm not sure why it would be have special or different... Note that the AWS Go SDK does have some env vars you may need to set, such as setting AWS_SDK_LOAD_CONFIG to true. Not sure if those make any difference?

[estebanz01]

👋 I'm having this problem too.

Terragrunt is not picking up the specified AWS profile that lives in ~/.aws/credentials. The problem i'm seeing is that terragrunt is using the IAM Role assigned to my EC2 dev instance and I need to run terragrunt in another AWS Account with the access key/secrets specified in another profile, not default (which is empty). Any ideas?

[domenjesenovec]

I can confirm adding [default] profile in ~/.aws/credentials solved the problem for me, thanks @khdevel

[eliz-ol]

I can confirm the confirmation from @domenjesenovec. It appears that the S3 backend requires the default profile in the plain text (ini style) %USERPROFILE%.awscredentials file. I tried it using a default profile in the the encrypted (SDK style) credentials file at %USERPROFILE%AppDataLocalAWSToolkitRegisteredAccounts.json and it failed with the same error mode.

The AWS Go SDK documentation seems to indicate that they expect the plain text file (about halfway down from the link that @brikis98 provided, under the header "Shared Credentials File"). It seems to me like the Go SDK makes a bunch of assumptions that aren't necessarily reasonable.