Terraform: InvalidClientTokenId while start plan

[birdyman]

provider "aws" {
        access_key = "access"
        secret_key = "secret"
        region = "ap-southeast-1"
}
resource "aws_instance" "example" {
        ami = "ami-32c3fd60"
        instance_type = "m3.large"
}

Just start and I got this error, the credential is full access and work to another script

[catsby]

Hello – could you please share the full error?
We've also release Terraform v0.6.3 in that time.
Are you using IAM credentials?

[jonDowdle]

I too get this error. We started with v0.5.3 and terraform plan works as expected (using 0.5.3) but when running plan using v0.6.3 I get the same error:

Refreshing Terraform state prior to plan...

Error refreshing state: 1 error(s) occurred:

* 1 error(s) occurred:

* InvalidClientTokenId: The security token included in the request is invalid.
        status code: 403, request id: [9800aa60-5d77-11e5-b968-83180be162d5]

[jantman]

I'm seeing the same issue with 0.6.3, using AWS IAM credentials leased from Vault (retrieved via vault-ruby, with terraform being executed from within a Rake task). When running terraform apply or plan, I get the 403 InvalidClientTokenId error, but if I use the same exact credentials (from Vault) with aws ec2 describe-instances, it works perfectly with no errors.

[jantman]

I'm relatively confident that the error _I_ am seeing is related to Vault, not TerraForm. My error appears to be due to https://github.com/hashicorp/vault/issues/687

[catsby]

Hello – is anyone still hitting this issue with the latest Terraform , version 0.6.6?

[zuriar]

Hi - I am hardcoding the AWS Credentials (not using Vault) and I am also getting this error using 0.6.6.

[catsby]

Hey @zuriar – you're using AWS credentials that are from IAM, correct? Is anyone using AWS_SESSION_TOKEN ? I'm not yet able to reproduce this

[zuriar]

Hi @catsby - I am using AWS creds, however I have just deleted the existing creds and created new ones. This time terraform plan/apply works fine, no error. Could it be a case of special characters in the credentials causing the error? Sometimes the creds have / \ ! etc. I am trying to replicate by creating new cred pairs. I'll let you know if the 'bug' re-appears, regards Hugh

[catsby]

Please do let us know, and thanks for following up!

I'm going to close this for now. Feel free to reply here if you find more info

[g-g-n]

GGs-MacBook-Pro:test ggn$ terraform -v
Terraform v0.6.6

GGs-MacBook-Pro:test ggn$ cat instance_aws.tf
provider "aws" {
access_key = "ACCESS_KEY_HERE"
secret_key = "SECRET_KEY_HERE"
region = "us-east-1"
}

resource "aws_instance" "example" {
ami = "ami-408c7f28"
instance_type = "t1.micro"

provisioner "local-exec" {
    command = "echo ${aws_instance.example.public_ip} > file.txt"
}

}
GGs-MacBook-Pro:test ggn$ terraform plan
Refreshing Terraform state prior to plan...

Error refreshing state: 1 error(s) occurred:

• 1 error(s) occurred:
• InvalidClientTokenId: The security token included in the request is invalid.
  status code: 403, request id: 43c37843-855b-11e5-a0a2-d9c97edbef9d
  GGs-MacBook-Pro:test ggn$

[nodefortytwo]

I had this error on 0.6.8 upgraded to 0.6.11 and its working now.

This is can be closed

[tyrostone]

@catsby I just upgraded to Terraform v0.6.15 and I am still seeing this issue. Code below:

provider "aws" {
    #region        = "${var.vpc_region}"
    region = "us-east-1"
    access_key = "ACCESS_KEY_HERE"
    secret_key = "SECRET_KEY_HERE"
}

resource "aws_s3_bucket" "service" {
    bucket = "dynarch-service-${var.environment}"
    acl = "private"
    versioning {
        enabled = true
    }

    tags {
        Name = "dynarch-service-${var.environment}"
        Environment = "${var.environment}"
        Product = "Dynamic"
    }
}

Command used to reproduce:

[16:18:41] lstone:buckets $ terraform plan -var-file=var-files/lapis_vars
Refreshing Terraform state prior to plan...

Error refreshing state: 1 error(s) occurred:

* 1 error(s) occurred:

* InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 8337af69-1235-11e6-be45-63f0a1aa55bb

I am able to reproduce this across two sets of AWS credentials, both of which contain \ or / in the secret key.

[aaron-bawcom]

+1

[cerdmann-pivotal]

+1

[Smuerdt]

+1

[kaizoku0506]

I'm seeing this error as well in 0.6.16.

Providers.tf file:

provider "aws" {
    region = "${var.aws_region}"
    access_key = "${var.secret_key}"
    secret_key = "${var.access_key}"
}

variables.tf file:

variable "aws_region" {
    default = "us-east-1"
}

variable "access_key" {}

variable "secret_key" {}

.envrc file:

export TF_VAR_access_key="access-key"
export TF_VAR_secret_key="secret-key"

error message received:

$ terraform plan
Refreshing Terraform state prior to plan...

Error refreshing state: 1 error(s) occurred:

* 1 error(s) occurred:

* InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: ...

I was able to resolve this on some systems by removing references to access_key and secret_key from providers.tf altogether. I was able to resolve it on other systems by moving access_key and secret_key to be the first two values under the aws provider in providers.tf.

[flynnCoolblue]

+1 also had this issue - fixed by re-generating credentials. appears to be an issue with certain chars.

[aqua0ps]

Same issue, on the very first time it work without any issue, and it stopped working next day, but if you recreate your credential it works.

I tried both by exporting the keys like export AWS_ACCESS_KEY_ID=

as well as declaring in the "terraform.vars"

example-app git:(master) ✗ terraform -v
Terraform v0.6.16

➜ example-app git:(master) ✗ terraform plan
Refreshing Terraform state prior to plan...

Error refreshing state: 1 error(s) occurred:

• 1 error(s) occurred:
• InvalidClientTokenId: The security token included in the request is invalid.
  status code: 403, request id: 6624a9ed-47cf-11e6-973a-d73c4a8360b1

[agentreno]

+1

[timxor]

+1000

[gabro]

Same here, on terraform 0.7.0

[avengerine]

Terraform v0.7.3, same issue.

_UPDATE_: Fixed, found weird chars in auth keys

[srjturner]

+1 Terraform v0.7.11 (it seems to be forward-slashes that trip it up)

[ptagr]

getting the same issue - Terraform v0.8.7
( no forward-slash in the creds)
providing creds using env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

[sherabi]

Still getting errors...

Terraform v0.9.2
Using credentials file with the following....

[nonprod]
aws_access_key_id = ACCESSKEY
aws_secret_access_key = SECRETACCESSKEY

provider "aws" {
  region = "${var.aws_region}"
  shared_credentials_file = "/Users/me/.aws/credentials"
  profile = "nonprod"
}

provider.aws: InvalidClientTokenId: The security token included in the request is invalid

[Yuxael]

I'm using terraform v0.9.5 and also encountered this problem. When I use ACCESS_KEY and SECRET_KEY stored in my ~/.aws/credentials it's fine, but when I use Vault-generated it returns

403 InvalidClientTokenId
The security token included in the request is invalid.

as previously mentioned.

data "vault_generic_secret" "aws_creds" {
  path = "aws/creds/role"
}

provider "aws" {
    region = "eu-central-1"
    access_key = "${data.vault_generic_secret.aws_creds.data["access_key"]}"
    secret_key = "${data.vault_generic_secret.aws_creds.data["secret_key"]}"
}

It happens only when creating new credentials.
I did some testing and I ran
'terraform plan
It failed with aforementioned error of course, but I dumped keyid and secret, put it in ~/.aws/credentials and ran it again. It worked.
It kinda looks like AWS had a delay and wasn't seeing these newly generated credentials for some reason.

[FlorinAndrei]

Seeing the same error like @Zerivael using vault-0.7.3 and terraform-0.9.11.

Would it be possible to introduce a delay after requesting the keys from Vault? Or repeat a few times if AWS authentication fails? It's an annoying error, and and old bug, surely it should be fixed by now.

[muhufuk]

+1

[apparentlymart]

Hi all,

Sorry for the ongoing trickiness in this area. Given the age of this closed issue and that the AWS provider is now developed in a separate repository it's be great if someone could open a fresh issue over there giving some updated details. (Things have changed a lot since 0.6.)

[FlorinAndrei]

@apparentlymart I suspect this ticket is related to the same problem, or a very similar one: https://github.com/terraform-providers/terraform-provider-aws/issues/1086

[Yuxael]

I don't know if it's related but is it possible that this is the cause (found in vault docs):

Unfortunately, IAM credentials are eventually consistent with respect to other Amazon services. If you are planning on using these credential in a pipeline, you may need to add a delay of 5-10 seconds (or more) after fetching credentials before they can be used successfully.

This information is under Dynamic IAM users section.

I switched to STS Federation Tokens and it's almost working (I've hit another issue) as they can be used right off the bat.

[FlorinAndrei]

@Yuxael But how do you add a delay in either Terraform or Vault?

And if this is indeed the problem, then Vault should wait until the credentials actually settle down. Or maybe run some queries to make sure everything is okay before handing the credentials over to other apps.

[FlorinAndrei]

@Yuxael Found it! It's definitely related to IAM keys eventual consistencies. I was able to introduce a delay in the Terraform pipeline, via a pretty horrible hack, since Terraform doesn't allow you a fine-grained-enough control over its execution to do simple things such as a delay.

This is the hack:

provider "vault" {
  address = "http://127.0.0.1:8200"
  skip_tls_verify = "true"
}

data "vault_generic_secret" "aws_iam_keys" {
  path = "aws/creds/admin"
}

data "external" "region" {
  # workaround for Vault bug
  # https://github.com/terraform-providers/terraform-provider-aws/issues/1086
  program = ["./delay-vault-aws"]
}

provider "aws" {
  region = "${data.external.region.result["region"]}"
  access_key = "${data.vault_generic_secret.aws_iam_keys.data["access_key"]}"
  secret_key = "${data.vault_generic_secret.aws_iam_keys.data["secret_key"]}"
}

And the corresponding delay-vault-aws script:

#!/usr/bin/env bash
sleep 10
echo '{ "region": "us-west-2" }'

Before these changes, I was never able to use Terraform with Vault-generated AWS IAM keys. It would always fail with:

provider.aws: InvalidClientTokenId: The security token included in the request is invalid.

After these changes, it works! I did terraform plan several times and it went through the whole list of queries and checks and planning and it never glitched out. While it's running, I see the newly generated vault-token users in IAM, which are then deleted. So everything seems normal.

Seems like I should really open a bug report with Vault instead.

[GeeH]

I can confirm that Terraform fails with the error

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

Error running plan: 1 error(s) occurred:

* provider.aws: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 87b2fa92-7c24-11e7-863c-cf1fe5f0ee40

only when the AWS secret has the special character \ in it.

[rberlind]

The workaround from @FlorinAndrei worked for me, although since I was using the hashistack Terraform module from https://github.com/hashicorp/nomad/tree/master/terraform/aws/modules/hashistack, I had to modify my Vault AWS policy to include both "ec2:" and "iam:" actions. Originally, I only had the "ec2:*" action. Here is my complete policy file:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1426528957000",
"Effect": "Allow",
"Action": [
"ec2:",
"iam:"
],
"Resource": [
"*"
]
}
]
}

[ozzielabrat]

Hi,

I am new to TerraForm and am working through the "TheTerraFormBook" by James Turnbull.

At the very first attempt to execute "terraform plan" I have struck this very same issue....it is certainly a bug.

Just installed 0.10.7 on an AWS EC2 instance running CentOS 7.4. This instance was created by me within the default AWS VPC (non-Classic).

I have an AWS user account with access_key and secret_key (and the secret_key contains characters like + and /).

Here's my testing:

When I create a very basic "base.tf" file as follows:

provider "aws" {
access_key = "obfuscated"
secret_key = "obfuscated_but_contains_a_+_and_a_/"
region = "ap-southeast-2"
}
resource "aws_instance" "base" {
ami = "ami-e9c7208b"
instance_type = "t2.micro"
}

Then I try and run "terraform plan", I get:

[dreeves@damions-terraform-instance base]$ terraform plan

Refreshing Terraform state in-memory prior to plan...

The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
Error running plan: 1 error(s) occurred:

* provider.aws: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 8c801ebb-b48b-11e7-ae61-3709d60281c4

If I then login to AWS Console and re-generate the AWS access_key and secret_key for my account (and end up with a secret_key that contains no + or / characters)....

Then I try and run "terraform plan" again it works successfully:

[dreeves@damions-terraform-instance base]$ terraform plan

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be

persisted to local or remote state storage.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:

• create

Terraform will perform the following actions:

• aws_instance.base
  id:
  ami: "ami-e9c7208b"
  associate_public_ip_address:
  availability_zone:
  ebs_block_device.#:
  ephemeral_block_device.#:
  instance_state:
  instance_type: "t2.micro"
  ipv6_address_count:
  ipv6_addresses.#:
  key_name:
  network_interface.#:
  network_interface_id:
  placement_group:
  primary_network_interface_id:
  private_dns:
  private_ip:
  public_dns:
  public_ip:
  root_block_device.#:
  security_groups.#:
  source_dest_check: "true"
  subnet_id:
  tenancy:
  volume_tags.%:
  vpc_security_group_ids.#:

Plan: 1 to add, 0 to change, 0 to destroy.

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Please resolve this bug....there are thousands of AWS accounts out there where the secret_key contains + and / characters....surely you cannot expect people to change all their keys ????

Thankyou,

Damion.

[khasim1227]

vi launch_insatance.tf

variable "region" {
type = "string"
default = "ap-south-1"
}
variable "akey" {
type = "string"
default = "XXXXXXXXXXXXXXXX"
(AWSAccessKeyId of your account)
}
variable "ckey" {
type = "string"
default = "XXXXXXXXXXXXXXXXXXX"
(AWSSecretKey of your account)

}
provider "aws" {
region = "${var.region}"
access_key = "${var.akey}"
secret_key = "${var.ckey}"
insecure = true
}
resource "aws_instance" "web" {
ami= "ami-d5c18eba"
instance_type= "t2.micro"
key_name= "test"
security_groups= ["mumbai"]
tags={
Name= "terraform-instance"
}
}

:wq!

./terraform plan

Enter a value: 1
./terraform apply

Now One New Server Lunch with Name Called "terraform-instance"
Best Regards
Khasim

[sgronblo]

I can confirm this bug. I had an aws_secret_access_key that included both a + and a \ and it failed with the "* provider.aws: InvalidClientTokenId: The security token included in the request is invalid." message.

I regenerated my access key until I got an aws_secret_access_key that consisted of only alphanumeric characters and it worked.

Terraform v0.11.3
+ provider.archive v1.0.0
+ provider.aws v1.8.0

[afedulov]

I am using aws-vault tool to pass AWS credentials and consistently get the same error. A workaround in this case is to disable sessions/tokens:
aws-vault exec <profile> --no-session -- terraform apply

[reza]

I can confirm a similar issue on latest version, in my case the value assigned to aws_profile was ignored and credentials in default profile were referenced that was invalid.

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: d506ee90-1578-11e8-8ba5-eb66686c63a9

$ terraform -v
Terraform v0.11.3
+ provider.archive v1.0.0
+ provider.aws v1.9.0
+ provider.local v1.1.0
+ provider.random v1.1.0
+ provider.template v1.0.0

[darshanchaudhary]

I was facing the same problem. Using a fresh new shell session solved the problem.

[Ovski4]

I had the same issue:

Error refreshing state: 1 error(s) occurred:

* module.module_name.provider.aws: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: ed7ae8d4-2ce8-11e8-b2c9-073a934138f8

If it can help anyone, what fixed this for me is to get rid of the alias property in the aws provider. That provider was nested in a module and not in the root files. I use an old version of Terraform (v0.9.11) so not sure if this apply to newer versions

[vaibhavmalushte]

upgraded to v0.11.5 still facing the same problem.
Kindly advise.

[DonBower]

+1
My Secret Access Key contains both the plus-sign (+) character, and multiple forward-slash (/) characters, including two in a row (blah/blah//foo/bar+/lunch)

[abadraja]

+1 on 0.11.7

[dawityifter]

+1 on Terraform v0.11.7
** provider.aws v1.19.0

[ajagarapu]

+1 on Terraform v0.11.7

• provider.aws: version = "~> 1.22"

[rozaydin]

+1

Terraform v0.11.7

• provider.aws v1.23.0

[vkaido]

+1

Terraform v0.11.7

• provider.aws v1.23.0
• provider.vault v1.1.0

[xyklex]

If it helps to anyone this is what works for me, after a couple hours investigating about it, I set up the next permissions to a dedicated IAM user.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket-name>",
                "arn:aws:s3:::<bucket-name/env/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetUser",
            ],
            "Resource": "<user-arn>"
        {
            "Effect": "Allow",
            "Action": [
                "sts:DecodeAuthorizationMessage",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

The content of my backend.tf

terraform {
  backend "s3" {
    encrypt = true
    bucket = "<bucket-name>"
    key    = "terraform.tfstate"
    region = "<bucket-region>"
    profile = "<aws-credential-profile>"
    workspace_key_prefix = "env"
  }
}

Also clean the file in .terraform/terraform.tfstate before doing terraform init again.

This worked for me using Terraform v0.11.7

[userhas404d]

Haven't run into the issue with invalid characters afaik but for those that are seeing this error when using STS temp credentials see: https://github.com/hashicorp/terraform/issues/6523
tl;dr: make sure you are setting AWS_SESSION_TOKEN.

[gyukebox]

+1

Terraform v0.11.7

• provider.aws v1.28.0

[mildwonkey]

Hi all,

Issues with the terraform AWS provider should be opened in the AWS provider repository.

Since this closed issue is still generating notifications for subscribers, I am going to lock it and encourage anyone experiencing issues with the AWS provider to open tickets there.

Please continue to open tickets here for any other terraform issues, and thanks!