Terragrunt: Feature Request: Use aws credentials profile

[jgartrel]

We have a common AWS account to store statefiles and manage locking for our terraform installations. It would be fantastic if you could allow the use of a specific "profile" in ~/.aws/credentials.

See: https://www.terraform.io/docs/state/remote/s3.html#profile

Configuration might look like this:

lock = {
  backend = "dynamodb"
  config {
    state_file_id = "my-app"
    aws_region = "us-east-1"
    table_name = "terragrunt_locks"
    max_lock_retries = 360
    profile = "terraform_state_prod"
  }
}

The credentials file might look like this:

[default]
aws_access_key_id = AKADADF
aws_secret_access_key = Wpaduaded32466

[terraform_state_prod]
aws_access_key_id = AKEASDFRE
aws_secret_access_key = zasdf3QWEEdae                                                                 

It would also be nice to specify the credentials file location, the default obviously being "~/.aws/credentials"

[jgartrel]

I don't know if it will help, but here is some documentation from AWS that may be relevant ....

http://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

"You can also use the SDK to select a profile by specifying os.Setenv("AWS_PROFILE", test-account) before constructing any service clients or by manually setting the credential provider, as shown in the following example":

sess, err := session.NewSession(&aws.Config{
    Region:      aws.String("us-west-2"),
    Credentials: credentials.NewSharedCredentials("", "test-account"),
})

[brikis98]

Have you tried using the AWS_PROFILE environment variable?

[josh-padnick]

To be fair, this is now the second time we've gotten this request. In https://github.com/gruntwork-io/terragrunt/issues/40 (the first request), I argued that the credentials file is unique to each developer and therefore embedding a profile name in the .terragrunt file doesn't promote the terragrunt's goal of "just working" when you terragrunt apply.

But I'm wondering if @jgartrel has standardized the AWS profile names across the team? If that's true, then each developer does in fact have a common set of profile names, and I can see why it'd be convenient to explicitly declare which profile you want.

Note also that this becomes especially useful in multi-account setups.

[jgartrel]

@brikis98 I have and I do use AWS_PROFILE, but it is non-optimal because I have to specify credential keys in our TF configs when they differ from the AWS_PROFILE ones

Usecase:

• Multiple AWS accounts
• All remote_state and terragrunt locking done in a centralized account
• terraform provisioning done in whatever account or region is necessary, using the developers default credentials or whatever they provide in env vars
• All developers are instructed to add a [terraform_state_prod] profile to their AWS credentials file with a shared set of credentials. This saves us from having to create developer accounts in every AWS account.

[jgartrel]

The plan was to have separate prod and stage profiles that would allow us to have separate remote_state data sources with environment specific shared info in them. This would allow us to not have our prod secrets exposed to all developers. But considering the vault provider just got merged down, we may not need to do this.

[josh-padnick]

All developers are instructed to add a [terraform_state_prod] profile to their AWS credentials file with a shared set of credentials. This saves us from having to create developer accounts in every AWS account.

Just an FYI, this is a very strong anti-pattern. Creating a unique IAM User account per developer is a pretty straightforward task, and means you no longer have to share keys. Keep in mind your security is generally only as strong as your weakest link, so using Vault to encrypt secrets would likely break down not because of Vault, but because a developer might leak their credentials, intentionally or otherwise.

Usecase: ...

Since Terraform supports the use of an optional profile property in the AWS provider, and because your use case makes sense to me, I would be in favor of adding a profile property (though probably in a different location in the file). @brikis98 Thoughts?

[brikis98]

So the use case is that everyone on the same team uses the same profile name in their ~/.aws/credentials file? Is that a common occurrence? Is that a practice we even want to encourage (as opposed to, say, using env vars)?

[jgartrel]

That is the case ONLY for remote_state and terragrunt locking. All other AWS terraform operations are done to the target account, using developer supplied credentials via ~/.aws/credentials or directly in the '.tf' file.

[jgartrel]

Would the team accept a pull request that looked something like this ... (Forgive me) as I have never programmed in GO before:

https://github.com/jgartrel/terragrunt/commit/c83f0f3b10a15970375d17923cd0766f47f1b518

[jgartrel]

developers are instructed to add a [terraform_state_prod] profile to their AWS credentials file with a shared set of credentials. This saves us from having to create developer accounts in every AWS account.

@josh-padnick , I Agree that it is an anti-pattern, and it was only a temporary workaround. But, the use case is still I think a valid one: To have a different AWS account in which you store locks, store remote_state and share remote_state. We have several deployments that span multiple AWS accounts and providers, and we expect users to use the central lock and remote state storage account.

In this account we can ensure the s3 bucket does encryption versioning, etc... and that state outputs can be shared amongst projects.

[brikis98]

@jgartrel I'm a bit concerned about using this profile only for the DynamoDB lock. What if you specify one profile in the .terragrunt file, but then Terraform itself uses a different profile because you specified the AWS_PROFILE env var or the profile parameter on the AWS provider. That opens the door to lots of confusion.

Relying on the env var keeps things simple and consistent. You could easily write a shell script for your team that wraps terragrunt with the proper profile:

export AWS_PROFILE=foo
exec terragrunt "$@"

Perhaps we could even update terragrunt to scan your Terraform code and find the profile parameter in the AWS provider, in which case your team could specify the provider in exactly one place. But if you're using different profiles for locks vs the rest of your infra, that won't work... But also strikes me as a rare use case...

[jgartrel]

@brikis98

So I was thinking there are 3 main AWS account scenarios for terragrunt:

1. Single AWS Account: You can specify your credentials in the [default] profile of ~/.aws/credentials or use the AWS_PROFILE environment variable to specify a different profile, but then why would you need to use profiles if you only had a Single AWS Account. Terragrunt (without changes) works perfectly for this scenario.
   
   
   
   • locks are stored in the single account
   
   
   • remote state is stored in the single account
   
   
   • resources are provisioned by terraform in the single account
   
   
2. Multiple Independent AWS Accounts: You can specify your credentials in the [default] profile of ~/.aws/credentials or use the AWS_PROFILE environment variable to specify a different profile. Each profile would represent a different account, and you are probably using a helper script to set AWS_PROFILE to the specific environment in which you are working.
   
   
   
   • locks are stored in the AWS_PROFILE account, this works, but this is not ideal since it sometimes tricky to hunt down stale locks.
   
   
   • remote state is stored in the centralized account using terraform remote state -backend-config="profile=central_account" since you are probably sharing outputs of terraform states by some external consumers, and you want all of that in one place
   
   
   • resources are provisioned by terraform in the AWS_PROFILE account
   
   
3. Multiple Inter-dependent AWS Accounts, or providers: In this scenario you are sharing outputs and have dependencies between resources in multiple AWS accounts or other providers. You want to keep your locks and remote centralized so you can depend on them and know where to look for them.
   
   
   
   • locks are stored in the centralized account using the [default] profile of ~/.aws/credentials. NOTE: this is very hard to enforce, and if you mess it up you might be running without solid locking and not even know it, until your statefiles are corrupt.
   
   
   • remote state is stored in the centralized account using terraform remote state -backend-config="profile=central_account" since you are definitely sharing outputs of terraform states and need to know where to find them
   
   
   • resources are provisioned by terraform in a specific profile referenced by an alias of the TF AWS provider, or they are provisioned by embedding credentials directly into configs through some data provider (like vault or consul). Under NO circumstances can you use the AWS_PROFILE env variable because it will either: redirect your locks to the wrong account, or it will override the embedded profile in your TF configs.
   
   

The crux of this issue is to give terragrunt users some options when they have multiple AWS accounts. Requiring users to have a specific account's credentials in the [default] section is hard to enforce and fails silently. Alternatively, requiring users to have a [tf_remote_state] section of their credentials file is very easy, and fails hard and fast with a clear error message.

[jgartrel]

Today we do use 'make' as a wrapper to terragrunt, and we do set AWS_PROFILE, however, that has led to a number of unintended side-effects for multiple account situations. Ideally, I do not want to store shared credentials in our TF repos and I don't want our AWS admins accidentally provisioning resources in the centralized production account mistakenly when they forgot to set a "profile" on their AWS providers in TF.

FWIW, I don't think it is wise to do "magic", such as scan the TF files for the "profile" section of the provider. I was rather looking for a simple and straight forward way to declaratively express it, and even require it.

[davidski]

Somewhat related, terragrunt currently also ignores the AWS_DEFAULT_REGION environment variable and hardcodes a default region of us-east-1 for the lock table. I expected TG to pull the default region out of my environment setting, just as credentials are today.

Does this make sense to include in this thread or is this worth a separate issue?

[brikis98]

@jgartrel Thanks for the detailed explanation! Now I understand your use case better. It makes sense and it seems like something we should support, so I'd welcome the PR you suggested. The only question is whether the profile parameter should be part of the lock entry only or part of the entire .terragrunt file...

[brikis98]

@davidski Please open a separate issue for that. The lock configuration does allow you to specify the region to use inside of the .terragrunt file, so it's not hard-coded to us-east-1, but it does not look for the AWS_DEFAULT_REGION env var, which is something that should be fixable with a PR. Thanks!

[allandrick]

Another vote for being able to provide the credentials for the DynamoDB access. I orchestrate across multiple AWS accounts and each of my terraform provider configs references specific credentials files stored outside the various repos (secured using Linux file ACLs today, with a view of moving to Vault in the future).

Being able to specify the shared_credentials_file would be great for dynamodb lock access configuration.

PS - we also use assume_role in our Terraform provider blocks to gain access to different accounts, which breaks when the environment variables are set (they take precedence in Terraform over credentials files).

[brikis98]

I believe this was fixed in #108 and released as part of v0.9.5.

[js-timbirkett]

FWIW - I wrap the running of Terragrunt in a rake task and have that export the correct environment variables and validate the account that's in use (actual vs expected) but we have a strange setup with multiple accounts per environment (2 x app cluster accounts and a "core" sidecar account) so orchestration is fun.