Terraform: InvalidClientTokenId: Terraform 0.9.2

[sherabi]

terraform --version
Terraform v0.9.2

Credentials file

[nonprod]
aws_access_key_id = ACCESS
aws_secret_access_key = SECRET

Provider

provider "aws" {
  region = "${var.aws_region}"
  shared_credentials_file = "/Users/me/.aws/credentials"
  profile = "nonprod"
 }

Error

Error refreshing state: 1 error(s) occurred:

* provider.aws: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 99c8662c-1943-11e7-bd65-978e35f3b40a

[catsby]

Hey @sherabi – is this an IAM user by chance, or perhaps running Terraform on an EC2 instance that has an Instance Profile ?

[sherabi]

@catsby It is a restricted IAM user. I am running terraform locally from my Mac. Hardcoding the access and secret keys inside the provider works just fine, but via a credentials file I get the error stated above. Unfortunately, its a bad idea to commit the access and secret keys into version control which is why I am going the credentials file route.

[catsby]

Thanks @sherabi – that's all reasonable, I was just trying to rule out a possible missing piece. The credential file should work, we'll investigate more.

Is there anything in your ~/.aws/config that would maybe be a clue here? Just trying to narrow down why/if _only_ using the key/secret in the provider block would be fine, but the cred file method is experiencing something with the token.

[sherabi]

No problem @catsby. Happy to give you any info that will help troubleshoot the issue.
The ~/.aws/config file has only this content

[default]
region = us-west-2

However it is over-ridden in the provider region = "${var.aws_region}"

[catsby]

Great, thank you! I was skeptical that config would have something, but I don't know it expertly so I wanted to rule it out.

[sherabi]

Hi @catsby I just wanted to run something by you. Is it possible that the security token is a mandatory requirement for using the credentials file? Based on the error in my first message it seems that may be the case... no?

[sherabi]

Hi @catsby I seem to have been able to get to the bottom of this issue.
You see if the "aws_secret_access_key" in the credentials file contains a "/" I seem to be getting the InvalidToken error from my original post.

Upon regenerating new credentials for my IAM user I made sure to keep generating one until the secret access key no longer had a "/" as part of its key. This seemed to have resolved the problem of the InvalidToken error.

I'll let you decide if you want to close this issue or keep it open given the workaround.

[sherabi]

@catsby Scratch my previous comment, it seems to have worked temporarily and now has stopped working again after I reloaded my tmux session.

[pioneerit]

Have the same problem:

$ env |grep AWS
AWS_DEFAULT_PROFILE=test
AWS_HOME=/Users/me/.aws
AWS_PROFILE=test

TF_LOG=TRACE (partitial): https://gist.github.com/pioneerit/6dd0dfa7bba53ae04e29a3aa690f99fc

$ cat  ~/.aws/config
[preview]
cloudfront = true

$ cat ~/.aws/credentials
[default]
# really like this, I don't like default so I put it
# otherwise aws-cli has bug to fail without any [default] profile
aws_access_key_id = some 
aws_secret_access_key = some

[test]
output = json
region = eu-central-1
aws_access_key_id = <hidden>
aws_secret_access_key= <hidden>



md5-293fba52cf8cd4dca0bc5854e0bf1a31



$ cat main.tf
provider "aws" {
  region = "${var.aws_region}"
  profile = "${var.aws_profile}"
  max_retries = "${var.max_retries}"
  allowed_account_ids = ["${var.aws_account_id}"]
}

terraform {
  required_version = "0.9.3"

  backend "s3" {
    profile = "test"
    encrypt = "true"
    bucket = "<hidden>"
    key = "<hidden>"
    region = "eu-central-1"
    lock_table = "<hidden>"
  }
}

resource "aws_s3_bucket" "test-some-123123123" {
  bucket = "test-asdfasdf-asdf-asdf-as-dfwe1"
}



md5-293fba52cf8cd4dca0bc5854e0bf1a31



$ terraform plan
...
* provider.aws: InvalidClientTokenId: The security token included in the request is invalid.
        status code: 403, request id

@catsby any ideas?

[pioneerit]

Found my problem 🤕
In my case, I was confused by TF Trace log. TF successfully get AWS access, even dynamo_db access was fine as weel.

But the actual code for the resources didn't work. Because I did a typo with TF_VAR_aws_profile

[adamhathcock]

I ran into this problem upgrading from 0.8 to 0.9.8 to use an S3 backend instead of the old S3 state management.

Trying to do a terraform init results in the same error as the OP. Simply not attempting to use the S3 backend leads to no problems.

How would my credentials be valid for everything else except for the S3 backend?

[adamhathcock]

Could be related to https://github.com/hashicorp/terraform/issues/14997

I'm trying to use eu-west-1

[khasim1227]

vi launch_insatance.tf

variable "region" {
type = "string"
default = "ap-south-1"
}
variable "akey" {
type = "string"
default = "XXXXXXXXXXXXXXXX"
(AWSAccessKeyId of your account)
}
variable "ckey" {
type = "string"
default = "XXXXXXXXXXXXXXXXXXX"
(AWSSecretKey of your account)

}
provider "aws" {
region = "${var.region}"
access_key = "${var.akey}"
secret_key = "${var.ckey}"
insecure = true
}
resource "aws_instance" "web" {
ami= "ami-d5c18eba"
instance_type= "t2.micro"
key_name= "test"
security_groups= ["mumbai"]
tags={
Name= "terraform-instance"
}
}

:wq!

[khasim1227]

write this type you will launch instance
Main error cause is in your AWSSecretKey Or AWSAccessKeyId .any one key has "/" symbol that ways we will get error .Declare above type the error was fix

[tiagocborg]

Thanks @khasim1227, you are a life saver.

[sverze]

Using a key without a '/' character does not fix the problem for me.

I am using terraform version 1.11.5 with aws provider 1.11.0 and I started to see this issue as of yesterday. Up until then I have not experienced any problems such as this.

I have tried 3 different accounts using various secret keys.
A few observations:

• I am able to create VPC resources with no problems
• This problem occurs when aws_instance resources are present
• The region does not matter
• I have tried changing AMI image identifiers as well

Any other ideas would be much appreciated

[vaibhavmalushte]

Facing same problem since last couple of days. I am on Terraform V0.11.5 & AWS provider v1.12.0.

Just trying to create VPC using https://github.com/terraform-aws-modules/terraform-aws-vpc

[sverze]

It's worth noting that I found the error goes away if you add the keys directly in to the provider section(s) for each module that creates EC2 resources.

provider "aws" {
access_key = ""
secret_key = ""
}

The above overrides any values you may have set in your local AWS credentials file.
This effectively kicks TF in to gear, I suspect that there is a caching problem.

Note that it only requires it once, you can remove the keys once TF is happy again.

[adaughterson]

@sverze I am finding that your solution only works as long as I hard-code the key/secret; it fails again once I remove them from my tf.
Any suggestions are most welcome. This all works perfectly well on my Mac, just failing on (of all things) Ubuntu.

[sverze]

I think I found the problem and solution.

It manifests when you have multiple profiles in your _.aws/credentials_ file which includes a default.
Terraform recommend that you do not have a _[default]_ credential, refer to https://github.com/terraform-providers/terraform-provider-aws.

If you still would like to use an AWS profile then you can do the following.
Add the _profile_ attribute to the _provider_ section:

provider "aws" {
  region        = "${var.aws_region}"
  profile       = "${var.aws_profile}"
}

When you run terraform from the command line specify the _aws_profile_ or add it as a variable:

terraform apply -var "aws_profile=your-profile"

[c4urself]

In my case I had to unset AWS_* env vars as they were invalid but still set.

[jdeprin]

In my case none of the above suggestions worked. The problem started after I rotated aws keys, my existing terraform builds would not accept my new keys. I build my plans with a -out=.terraform/tfplan flag. Deleting the .terraform directory and then re-running a terraform init solved my specific issue.

[mysysadmin-ltd]

The only thing that worked for me (I didn't try hardcoding the keys), was to rename my profile in ~/.aws/credentials to "default".

[jaamo]

For me generating the secret without / -character worked.

[colde]

Basically what @mysysadmin-ltd said. It seems that the s3 backend doesn't use the profile defined in the provider "aws" block. For some reason it only looks at the default profile.

[davidjeddy]

Confirming @colde 's findings.

Terraform 0.11.11 remote state logic seems to use the ~/.aws/credentials [default] credentials and not the credentials provided via the provider.tf configuration.

This one took me about an hour to sort out. One can reasonably assume developers may want to keep credentials isolated to an applications config OR have multiple sets of credentials in ~/.aws/credentials.

[dave2535]

Terraform v0.11.11

having AWS secret key without a / fixed this issue for me on March 6, 2019

my ~/.aws/credentials file includes a default and 2 other profiles. The first new secret generated today contained a /. Had to regenerate several times before AWS gave one without a /. Once that was put into place, terraform init ran successfully

[YoninL]

After remove .terraform folder and redo init it works, must be some caching problem as this happens when I rotate my AWS key.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.