Terraform-provider-aws: AWS provider fails with InvalidClientTokenId

[hashibot]

_This issue was originally opened by @bitemyapp as hashicorp/terraform#5955. It was migrated here as part of the provider split. The original body of the issue is below._

---

https://gist.github.com/bitemyapp/f88fcdff2c1d701f6affdcd55c9691d5

Error refreshing state: 1 error(s) occurred:

* 1 error(s) occurred:

* InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: caa59160-f75f-11e5-9241-eb1509222747

Following along with the simplest AWS example y'all had in the repository.

Makefile

# AWS_ACCESS_KEY_ID=123 AWS_SECRET_ACCESS_KEY=456
# AWS_ACCESS_KEY_ID=123 AWS_SECRET_ACCESS_KEY=456 terraform plan
# AWS_ACCESS_KEY_ID=123 AWS_SECRET_KEY=456 terraform plan
# AWS_ACCESS_KEY=123 AWS_SECRET_KEY=456 terraform plan -var-file="terraform.tfvars"

plan:
    terraform plan -var-file="terraform.tfvars"

variables.tf

variable "aws_region" {
  description = "AWS region to launch servers."
  default = "us-west-2"
}

# Ubuntu 14.04 LTS (x64)
variable "aws_amis" {
  default = {
    us-west-2 = "ami-8ba74eeb"
  }
}

variable "aws_pem_key_file_path" {
  description = "Path to the AWS pem key"
}

variable "aws_key_name" {
  description = "AWS pem key name"
}

variable "aws_access_key" {
  description = "Access key to provider (AWS, openstack, etc)"
}

variable "aws_secret_key" {
  description = "Secret key to provider (AWS, openstack, etc)"
}

main.tf

# Specify the provider and access details
provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region = "${var.aws_region}"
}

# Create a VPC to launch our instances into
resource "aws_vpc" "default" {
  cidr_block = "10.0.0.0/16"
}

# Create an internet gateway to give our subnet access to the outside world
resource "aws_internet_gateway" "default" {
  vpc_id = "${aws_vpc.default.id}"
}

# Grant the VPC internet access on its main route table
resource "aws_route" "internet_access" {
  route_table_id         = "${aws_vpc.default.main_route_table_id}"
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = "${aws_internet_gateway.default.id}"
}

# Create a subnet to launch our instances into
resource "aws_subnet" "default" {
  vpc_id                  = "${aws_vpc.default.id}"
  cidr_block              = "10.0.1.0/24"
  map_public_ip_on_launch = true
}

# A security group for the ELB so it is accessible via the web
resource "aws_security_group" "elb" {
  name        = "terraform_example_elb"
  description = "Used in the terraform"
  vpc_id      = "${aws_vpc.default.id}"

  # HTTP access from anywhere
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # outbound internet access
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Our default security group to access
# the instances over SSH and HTTP
resource "aws_security_group" "default" {
  name        = "terraform_example"
  description = "Used in the terraform"
  vpc_id      = "${aws_vpc.default.id}"

  # SSH access from anywhere
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # HTTP access from the VPC
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  # outbound internet access
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}


resource "aws_elb" "web" {
  name = "terraform-example-elb"

  subnets         = ["${aws_subnet.default.id}"]
  security_groups = ["${aws_security_group.elb.id}"]
  instances       = ["${aws_instance.web.id}"]

  listener {
    instance_port     = 80
    instance_protocol = "http"
    lb_port           = 80
    lb_protocol       = "http"
  }

}

resource "aws_key_pair" "auth" {
  key_name   = "${var.aws_key_name}"
  public_key = "${file(var.aws_pem_key_file_path)}"
}

resource "aws_instance" "web" {
  # The connection block tells our provisioner how to
  # communicate with the resource (instance)
  connection {
    # The default username for our AMI
    user = "ubuntu"

    # The connection will use the local SSH agent for authentication.
  }

  instance_type = "m1.small"

  # Lookup the correct AMI based on the region
  # we specified
  ami = "${lookup(var.aws_amis, var.aws_region)}"

  # The name of our SSH keypair we created above.
  key_name = "${aws_key_pair.auth.id}"

  # Our Security group to allow HTTP and SSH access
  vpc_security_group_ids = ["${aws_security_group.default.id}"]

  # We're going to launch into the same subnet as our ELB. In a production
  # environment it's more common to have a separate private subnet for
  # backend instances.
  subnet_id = "${aws_subnet.default.id}"

  # We run a remote provisioner on the instance after creating it.
  # In this case, we just install nginx and start it. By default,
  # this should be on port 80
  provisioner "remote-exec" {
    inline = [
      "sudo apt-get -y update",
      "sudo apt-get -y install nginx",
      "sudo service nginx start"
    ]
  }
}

$ terraform -v
Terraform v0.6.14

What am I doing wrong?

[ujann]

I have a similar problem with terraform v0.10.8 in our CI:
i am deploying a Lambda function with a role, and during a terraform plan, terraform tries to do a Action=GetRole&RoleName=lambda_role-worker;
this all works with "real" credentials (access key & secret key of the CI user);
but running in our CI, the script gets temporary credentials for that same CI user (access key, secret key + session token); most aws commands work, but when terraform comes to the point when it tries to check the role, it fails with a
aws_iam_role.worker-lambda_role: aws_iam_role.worker-lambda_role: Error reading IAM Role lambda_role-worker: InvalidClientTokenId: The security token included in the request is invalid status code: 403
I guess this is because of You cannot call any IAM APIs unless MFA authentication information is included in the request. (http://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html)

Is there a way around this?

[ujann]

To answer my own question:
The only way to get temporary session credentials that have the right to execute IAM calls, is to obtain them via sts:AssumeRole; session creds obtained via GetFederationToken or GetSessionTokenwill always fail with "IAM:*" calls;

to create credentials that can execute assume-role, the owning IAM user must have the right to do this for the role ({"Effect": "Allow", "Action": "sts:AssumeRole","Resource": "arn:aws:iam::<accountId>:role/<role-name>"}) and the role must have the AWS account itself as the trust Entity ({ "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<accountId>:root" }, "Action": "sts:AssumeRole" }).
The resulting temporary credentials then have the same rights as defined in the role

Bamboo's "Identity Federation Plugin" supports the option "Assume Role", so this is the way we go for us.

[Ipswitch]

What's the status?
I'm using a provider pulled today and received
"provider.aws: InvalidClientTokenId: The security token included in the request is invalid"

I've no issues using get-session-token and calling IAM functions with the CLI.
aws iam add-user-to-group --group-name admin --user-name bill

[frzhanguber]

ujann, I think I ran into the same issue as yours, and the rood cause I believe is as you mentioned: You cannot call any IAM APIs unless MFA authentication information is included in the request

Even I made the change as yours, I am still getting the same error, my assumption is: I use a federated account, and it only works for GetSessionToken not for GetFederationToken, since GetFederationToken "Cannot call IAM APIs directly."

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!