Terraform-provider-aws: Terraform seems to be ignoring skip_credentials_validation flag for AWS provider

[hashibot[bot]]

_This issue was originally opened by @Ghazgkull as hashicorp/terraform#18696. It was migrated here as a result of the provider split. The original body of the issue is below._

---

We are trying to use Terraform to create AWS resources in LocalStack. It works correctly if we first authenticate to AWS, but fails when lacking AWS credentials.

Since we are pointing Terraform at LocalStack and not the real AWS, there's no need for Terraform to authenticate to AWS. So we are settings both skip_credentials_validation and skip_metadata_api_check to true. Various blog posts on the internet suggest this as the way to point Terraform to LocalStack, so it presumably worked at some point?

Terraform Version

Terraform v0.11.8

Terraform Configuration Files

provider "aws" {
  region                      = "${var.region}"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  s3_force_path_style         = true
  access_key                  = "mock_access_key"
  secret_key                  = "mock_secret_key"

  endpoints {
    dynamodb = "http://localhost:4569"
    s3       = "http://localhost:4572"
  }
}

Expected Behavior

When an AWS provider is configured with skip_credentials_validation = true, credential validation is skipped.

Actual Behavior

Terraform still tries to validate credentials.

Steps to Reproduce

Run terraform plan with an AWS provider configured to skip credential validation.

Additional Context

Terraform output at the time of credential validation:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: Failed getting account information via all available methods. Errors: 2 errors occurred:
    * error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 1685939e-a192-11e8-b421-e51413e3548d
    * failed getting account information via iam:ListRoles: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 16c2761c-a192-11e8-8c67-63bb9d86555a

[mberlanda]

It looks like a breaking change has been introduced in v1.31:https://github.com/terraform-providers/terraform-provider-aws/commit/e56ded8f5b032187fdfc3f8df7a158fc65238aa3

Forcing v1.30 will work.

my guess is something messing with:

provider: Allow provider configuration AssumeRoleARN and sts:GetCallerIdentity credential validation call to shortcut account ID and partition lookup (#5177)

Would it be possible to put this credential validation behind skip_credentials_validation flag?

[bflad]

Using skip_requesting_account_id = true in your provider configuration should workaround this issue:

Skip requesting the account ID. Useful for AWS API implementations that do not have the IAM, STS API, or metadata API. When set to true, prevents you from managing any resource that requires Account ID to construct an ARN.

Previously when fetching account ID information, the STS call ignored all errors:

https://github.com/terraform-providers/terraform-provider-aws/blob/c09b938476f81d517c448b59fbbc9dfb94c52bf3/aws/auth_helpers.go#L70-L77

I will submit a pull request to reinstate that behavior.

[bflad]

After looking more at the old code, it was actually that we completely ignored errors when trying to find account ID, not specifically that we were ignoring any of the specific calls.

https://github.com/terraform-providers/terraform-provider-aws/blob/c09b938476f81d517c448b59fbbc9dfb94c52bf3/aws/config.go#L379-L384

While we can reinstate this behavior since changing the provider initialization process is considered a major breaking change, please note that in the next major version may require skip_requesting_account_id = true again. Its important for operators to understand the implications of the provider account ID not being available/correct for certain data sources and resources, e.g. #5793

[bflad]

Summary:

If the provider is unable to determine the AWS account ID from a provider assume role configuration or the STS GetCallerIdentity call used to verify the credentials (if skip_credentials_validation = false), it will attempt to lookup the AWS account ID via EC2 metadata, IAM GetUser, IAM ListRoles, and STS GetCallerIdentity. Previous to version 1.31.0 of the provider, it would silently allow the failure of all the above methods. We inadvertently changed this behavior to return an error since seemed more correct in the context of the other changes occurring at the time.

There are implications of the provider not having its AWS account ID properly initialized, which are noted at: https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id

Since the provider behavior change was breaking, I have submitted a pull request to temporarily reinstate the old behavior and note the upcoming error change in the next major version of the provider: #5794

I would recommend adding the following to your provider configurations, which should work with the latest 1.X versions (even before the above pull request is released) as well as 2.X:

```hcl
provider "aws" {
# ... other configuration ...

skip_requesting_account_id = true
}
```

[Ghazgkull]

@bflad FYI, I tried both of the workarounds suggested here and neither worked.

For reference, here's the provider configuration I tried:

provider "aws" {
  version                     = "~> 1.30.0"
  region                      = "${var.region}"
  skip_requesting_account_id  = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  s3_force_path_style         = true

  endpoints {
    dynamodb = "http://localhost:4569"
    s3       = "http://localhost:4572"
  }
}

And here's the output I saw on the console:

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.30.0)...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Workspace "local" doesn't exist.

You can create this workspace with the "new" subcommand.
Created and switched to workspace "local"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

Error: Error running plan: 1 error(s) occurred:

* provider.aws: No valid credential sources found for AWS Provider.
    Please see https://terraform.io/docs/providers/aws/index.html for more information on
    providing credentials for the AWS Provider


Failed to deploy terraform. Exiting.

[bflad]

@Ghazgkull I believe you may also need to provide fake access key and secret key credentials, either via the AWS credentials file or provider configuration.

We should probably create a how-to guide for configuring the provider for LocalStack. 😄

[Ghazgkull]

Yes. With the latest AWS provider (version 1.41), the following config works to run Terraform against LocalStack without actually authenticating to AWS:

provider "aws" {
  region                      = "${var.region}"
  skip_credentials_validation = true
  skip_requesting_account_id  = true
  skip_metadata_api_check     = true
  s3_force_path_style         = true
  access_key                  = "mock_access_key"
  secret_key                  = "mock_secret_key"

  endpoints {
    dynamodb = "http://localhost:4569"
    s3       = "http://localhost:4572"
  }
}

I think it's worth noting that adding a fake access_key and secret_key feels like a workaround to a defect with the flags. It would be nice if the flags just worked without needing additional properties provided.

[arinzemomife]

I am having this problem too cant seem to find a solution. Any help guys?

[isaccavalcante]

I had the same problem, but in my case it was because I was not specifying the ec2 endpoint in the configuration, as I copied from this example.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!