Terraform: "AWS Provider not being passed to child modules" or "Terraform silently uses the default AWS credentials" or "Terraform prompts to enter provider.aws.region"

[martin-flaregames]

The title of this ticket is basically the summary of the 3 observable problems that may occur as a chain reaction when nesting modules in a more or less complex way.

I stumbled over this issue in terraform 0.6.x and pin pointed the cause in terraform 0.7.2, when sometimes terraform prompted me to manually enter a value for provider.aws.region, at first I ignored and manually entered the region again, and mostly several times per terraform plan and terraform apply runs. This was an mistake, just cancel the operation when this occurs.

In reality terraform 1) failed to pass the configured AWS provider to the used sub modules and 2) internally was falling back to the default AWS credentials, which in my case differ from the ones I used in terraform. So all changes were applied in my AWS account and not in the AWS account of my customer.

Terraform Version

v0.8.0 to v0.8.7 (older affected versions include v0.6.x, v0.7.x)

Affected Resource(s)

• AWS provider?
• Terraform core magic?

Terraform Configuration Files

Checkout https://github.com/martin-flaregames/terraform-aws-region-prompt and follow the instructions provided below.

Expected Behavior

The provider should be correctly inherited

Actual Behavior

Somewhere in the module loading chain the provider config is lost and the default credentials are used, and most of the time this is only detected when terraform prompts to manually enter a provider.aws.region in the terminal

Steps to Reproduce

in order to reproduce this issue check out https://github.com/martin-flaregames/terraform-aws-region-prompt and follow this instructions.

you will need to configure your default credentials in ~/.aws/credentials and do not provide any default region information

[default]
aws_access_key_id=YOURACCESSKEY001
aws_secret_access_key=YOURSECRETKEY001

then create a terraform.tfvars file and provide different credentials as the ones used in your ~/.aws/credentials file, use it with the previously provided github example, copy the file into the example1 and example2 folders.

aws_region = "us-east-1"
aws_access_key = "YOURACCESSKEY002"
aws_secret_key = "YOURSECRETKEY002"
````
#### working example (example1 folder)

open a bash shell and execute the following commands
```hcl 
cd example1
terraform get
terraform plan

you should get a execution plan as expected.

+ module.vpc.vpc.aws_vpc.vpc
    cidr_block:                "10.10.0.0/19"
    default_network_acl_id:    "<computed>"
    default_route_table_id:    "<computed>"
    default_security_group_id: "<computed>"
    dhcp_options_id:           "<computed>"
    enable_classiclink:        "<computed>"
    enable_dns_hostnames:      "true"
    enable_dns_support:        "true"
    instance_tenancy:          "<computed>"
    main_route_table_id:       "<computed>"
    tags.%:                    "1"
    tags.Name:                 "terraform1-vpc"

failing example (example2 folder)

open a bash shell and execute the following commands

cd example2
terraform get
terraform plan

the plan will not run but the terraform AWS provider will ask for provider.aws.region

provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Default: us-east-1
  Enter a value: 

Interestingly I have the impression that this occurs when at least 2 child modules do not use any AWS resources but only include other sub modules.

The module chain in the provided example2

I just provide the terminal output of repeating cat *.tf and cd <dir> using the module relative path, this way you can better see the module chaining and the contents of the used terraform files.

terraform-aws-region-prompt/example2 (master)  cat *.tf

#
# This is failing example based on the working example 1
#

# :: init
variable "aws_access_key" {} # terraform.tfvars
variable "aws_secret_key" {} # terraform.tfvars
variable "aws_region" {} # terraform.tfvars

provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region = "${var.aws_region}"
}

# :: code
module "example2_failing" {
  source = "../modules/example2_failing"
}



md5-bd61d9399356fe9788359c48cacb5b6e



terraform-aws-region-prompt/example2 (master)  cd ../modules/example2_failing
terraform-aws-region-prompt/modules/example2_failing (master)  cat *.tf



md5-bd61d9399356fe9788359c48cacb5b6e



#
# This is basically a copy of example1 but as module
#

variable "stackname" { default = "terraform2" }

module "vpc" {
  source = "../../modules/devops/vpc/autoaddr"
  stack_name = "${var.stackname}"
}



md5-bd61d9399356fe9788359c48cacb5b6e



terraform-aws-region-prompt/modules/example2_failing (master)  cd ../../modules/devops/vpc/autoaddr
terraform-aws-region-prompt/modules/devops/vpc/autoaddr (master)  cat *.tf



md5-bd61d9399356fe9788359c48cacb5b6e



variable "stack_name" {}

module "vpc" {
  source = "../"
  stack_name = "${var.stack_name}"
}



md5-bd61d9399356fe9788359c48cacb5b6e



terraform-aws-region-prompt/modules/devops/vpc/autoaddr (master)  cd ../
terraform-aws-region-prompt/modules/devops/vpc (master)  cat *.tf



md5-bd61d9399356fe9788359c48cacb5b6e



variable "stack_name" {}

resource "aws_vpc" "vpc" {
  cidr_block = "10.10.0.0/19"
  enable_dns_support = true
  enable_dns_hostnames = true
  tags {
    Name = "${var.stack_name}-vpc"
  }
}

References

8680

[mitchellh]

Ahhhh I got it reproduced.

This is annoying because it works in the new graphs (plan, refresh, apply) but is being blocked by the legacy graphs which are still defaulted on in input and validate. In particular, you're being asked in the input step.

This will be fixed by TF 0.9 when we redo the input/validate graphs to the newer graphs, which already have the transforms to fix this, embarassingly.

[martin-flaregames]

@mitchellh good to know that you can reproduce the problem on you side. I started wondering if it was only me (and my colleagues).
I see you say:

...but is being blocked by the legacy graphs which are still defaulted on in input and validate.

Am I missing something there? Is there a way to tell terraform 0.8 to use the new graphs using some(tm) magic(tm) command line arguments?

[mitchellh]

Hey @martin-flaregames. The "input" and "validate" operations don't have new graphs at all yet so there is no magic switch to enable them. :( Frustratingly, they're the easier graphs to implement (we did the hard ones first: plan, apply, destroy), so its not much work and we'll definitely get it done before 0.9.

[martin-flaregames]

... so its not much work and we'll definitely get it done before 0.9.

That would be great.

[donnoman]

It's affecting me and I only have a single module, and the region is defined IN the module

in the modules/site/site.tf

variable "aws_region" {
  default = "us-west-2"
}

provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region = "${var.aws_region}"
}

in the main terraform

module "site" {
  source = "../modules/site"
  stage  = "${var.stage}"
  aws_amis = "${var.aws_amis}"
  aws_access_key="${var.aws_access_key}"
  aws_secret_key="${var.aws_secret_key}"
  aws_rds_username="${var.aws_rds_username}"
  aws_rds_password="${var.aws_rds_password}"
}

the aws_ variables are all set by terraform.tfvars

I don't want to pass a region from here because I intend the terraform to do multi-region actions. Right now I just need it to stop prompting me.

[donnoman]

this is just nutty

provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region = "us-west-2"
}

provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Default: us-east-1
  Enter a value: ^C

› terraform -version
Terraform v0.8.2

sorry... thats gotta get fixed.

[martin-flaregames]

This is giving me nightmares. I keep hitting this wall very often when using terraform with more advanced module strategies :(

... so its not much work and we'll definitely get it done before 0.9.

Any time estimates?

Many thanks in advance!

[mitchellh]

I'm going to take a look at this today. Sorry for the delay.

[AnthonyNeace]

In 0.8.7 I'm seeing similar behavior in child modules when passing the profile argument while also having environment variables set for my dev AWS account Access and Secret Keys:

printenv
...
AWS_SECRET_ACCESS_KEY=mydefaultDevSecretKey
AWS_ACCESS_KEY_ID=mydefaultDevAccessKey

Provider example:

provider "aws" {
    region = "us-east-1"
    profile = "prod"
}

I have [default] and [prod] profiles set in my AWS credentials file for AWS CLI. After unsetting these environment variables, my child modules began respecting the prod profile configured in my top-level .tf file's aws provider.

[kamsz]

Experiencing the same issue with 0.8.4.

[mitchellh]

Ah, so the bug was actually fixed by #11426.

I can reproduce this bug on 0-8-stable but no longer on master. This means it affects all of 0.8.x but not 0.9.0 (unreleased). Unfortunately #11426 is not able to be backported and the fix onto 0-8-stable directly is non-trivial. The good news is that we're already in beta of 0.9 (beta 1 was yesterday) and we plan to release 0.9 mid-March with RCs starting by Mar 1.

My recommendation then is to use 0.9. If you're unsure about a beta (you should definitely be unsure for production infrastructure), then waiting for the RCs is a good idea and only a couple weeks away.

Sorry about all this I know its affecting a lot of people but given we're so close at this point and we already have binaries out, that's what I recommend!

[martin-flaregames]

Looking forward for Terraform 0.9!

[youssefNM]

i still can reproduce this issue with terraform version 0.9.6 !!!

[catherinetcai]

Reproducing this issue on 0.10.0 as well.

[malbiez]

Same here, issue still present in Terraform v0.10.0.

[bruce-davis]

Same issue in Terraform v0.10.2

[goldelehr]

Same issue in Terraform v0.10.5

[jniedrauer]

Same issue in Terraform v0.10.6

[aliusmiles]

I've got similar issue, fixed it by using another tfstate file.
Not sure if it's related, but here is myterraform providers output while having issue:

$ terraform providers
.
├── provider.aws.local (from state)
├── module.global
│   ├── provider.aws.local

And after using new tfstate file (different s3 key):

$ terraform providers
.
├── module.global
│   └── provider.aws.local

[alevinetx]

And 10.7

[tjausten]

Happening in 0.11.0

[apparentlymart]

The module/provider interaction was completely rewritten for Terraform 0.11, so if you are still seeing problems I'd ask that you open a fresh issue and fill out the issue template since it's very unlikely that the cause is the same even if the symptoms seem similar, and so having new versions of all of the requested info in the issue template will be required to dig in and debug.

[aliusmiles]

I suspect this issue is not related to terraform version, but rather to old resources in state file.
I had resources with aliased provider in two folders, and then moved resources into one folder:
before:

.
├── bar
│   └── bar.tf
├── foo
│   └── foo.tf
└── main.tf

after:

.
├── foo
│   ├── bar.tf
│   └── foo.tf
└── main.tf

terraform began prompting region value, even though all resources still have properly configured provider. I've destroyed everything, and apply from scratch -- no prompting anymore. Second time this happened, I've just input region name, and after successful apply it has never asked for region again.

I didn't have time to investigate this deeper, but I guess when you delete/rename resources in *.tf files, but they are still present in state file, terraform just falls back to default provider configuration for them, and if you don't have one -- that's when prompt appears.

[tjausten]

Thanks @aliusmiles for the follow-up. I do have a default provider configuration that is included with the prompt.

provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Default: us-east-1
  Enter a value: 

In addition, I am using a Makefile with the Terraform commands included. That could be causing my issue. I will take your feedback and investigate further.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.