Terraform: Child-module-specific Provider Configuration

[apparentlymart]

Consider the following root module:

module "child-usw2" {
  source = "./child"

  region = "us-west-2"
}

module "child-use1" {
  source = "./child"

  region = "us-east-1"
}

...and the following ./child:

variable "region" {
}

provider "aws" {
  region = "${var.region}"
}

resource "aws_instance" "example" {
  # ...
}

When this configuration is applied, it will (as expected) create one EC2 instance in the us-west-2 region and another in the us-east-1 region, and record both of these in state.

If the user then removes the module.child-use1 block from the root module and re-plans, we run into a problem: Terraform no longer has the provider "aws" block from that module, so there's not enough information to destroy module.child-use1.aws_instance.example. In this case, Terraform will fail because the region argument is required for the aws provider.

Working around this currently requires some awkward steps:

• terraform destroy -target=module.child-use1
• _then_ remove the child module block from the root
• terraform plan should produce an empty plan, since the resources were already destroyed

---

Confronted with this issue, some users then try to hoist the provider declarations up to the root:

provider "aws" {
  region = "us-west-2"
  alias  = "usw2"
}

provider "aws" {
  region = "us-east-1"
  alias  = "use1"
}

module "child-usw2" {
  source = "./child"

  aws_provider = "aws.usw2"
}

module "child-use1" {
  source = "./child"

  aws_provider = "aws.use1"
}

and then in the child module:

variable "aws_provider" {
}

resource "aws_instance" "example" {
  # ...

  provider = "${var.aws_provider}"
}

This fails, because Terraform does not permit dynamically-populating the provider pseudo-argument in this way. (It can't, because that value is needed for graph construction.)

---

Being able to instantiate the same module multiple times with different provider settings is useful in a number of situations, including the above example of AWS regions. It'd be good to find a different config formulation that avoids the usability problems in the first example without creating the chicken-and-egg problems that'd result from supporting the latter.

Some related issues:

• #1819 was a broad issue covering a number of weird behaviors around inherited modules, some of which have now been addressed. This issue is a more-specific offshoot of that one.
• #4149 describes a mechanism that allows us to address certain classes of chicken and egg problem, and may contribute to a solution here though I'd consider this a path of last resort.

[badmojo76]

We are running into this exact issue, using scenario 1 (passing a region variable into the module). While digging into it, I noticed that the terraform.tfstate contains the provider as a variable, not a static value:

"aws_eip.nat_eip.1": {
    "type": "aws_eip",
    "depends_on": [],
    "primary": {
        "id": "eipalloc-98765432",
        "attributes": {
            "association_id": "",
            "domain": "vpc",
            "id": "eipalloc-98765432",
            "instance": "",
            "network_interface": "",
            "private_ip": "",
            "public_ip": "34.45.56.67",
            "vpc": "true"
        },
        "meta": {},
        "tainted": false
    },
    "deposed": [],
    "provider": "aws.${var.target_region}"
},

If these variables were resolved and saved in the state file (or if there was a region section in the provider definition in state), the removal of the module doesn't require the user to enter a region (and with more complex configurations, if multiple modules across regions are removed simultaneously, it avoids the corrupted state file that mismatches the AWS configuration).

The only caveat to this is that it requires a hybrid of scenario 1 AND scenario 2, above...

# These providers are used when the module is destroyed by removing it.  
# Since the state file would have a static value (aws.us-east-1, 
# not aws.${var.target_region}), there is no need to specify a region, 
# since the state KNOWS where the resources are deployed (which it
# SHOULD know anyway, right?)

provider "aws" {
  region = "us-west-2"
  alias  = "us-west-2"
}

provider "aws" {
  region = "us-east-1"
  alias  = "us-east-1"
}

module "child-usw2" {
  source = "./child"
  target_region = "us-east-2"
}

module "child-use1" {
  source = "./child"
  target_region = "us-east-1"
}

...and in the ./child module:

variable "target_region" {
}

# This provider is configured to allow for re-usability of the module across multiple regions
provider "aws" {
  region = "${var.target_region}"
}

resource "aws_vpc" "default" {
  provider   = "aws.${var.target_region}" <--- Resolve this before inserting into terraform.tfstate!
  cidr = "10.0.0.0/16"
...
}

To test this, you can setup the config above with a simple resource, create it, edit the terraform.tfstate and replace the aws.${var.target_region} with aws.us-east-1 for child-use1 and aws.us-west-2 for child-usw2. Then, delete/comment out the child-use1 or usw2 module (or both) and everything cleans up nicely without a prompt.

It seems to me that the state file should have a static knowledge of where resources exist in the infrastructure - there are no other places in the terraform.tfstate that I've seen or been able to create that store an embedded variable, which seems like the heart of this issue.

[sshvetsov]

Beating my head against this issue as well while trying to create AWS Config recorders in every region using v0.9.11.

I'm defining alias and provider in the module definition:

# modules/tf_config/main.tf
provider "aws" {
  region = "${var.aws_region}"
  alias  = "${var.aws_region}"
}

resource "aws_config_configuration_recorder" "recorder" {
  provider = "aws.${var.aws_region}"
  name     = "${format("recorder-%s", var.aws_region)}"
  ...
}
...

Then passing the region variable when instantiating teh module:

# main.tf
module "config_us_east_1" {
  source           = "../modules/tf_config"
  aws_region       = "us-east-1"
  ...
}

module "config_us_east_2" {
  source           = "../modules/tf_config"
  aws_region       = "us-east-2"
  ...
}

Creation works, but removing module instantiation code wouldn't destroy the resources.

I played around with manual state changes, like @badmojo76 suggested, and I agree that interpolating the variables used in specifying resource provider in the state seem to solve the problem.

[rcrelia]

+1 for having static-via-interpolation provider regions in the state file

[apparentlymart]

Hi all! Thanks for sharing your configs and use-cases.

The Terraform team is currently planning a collection of changes to the handling of modules and, amongst other things, their interactions with providers. Since there are many somewhat-related issues here that all affect similar portions of the code, we're approaching it holistically to try to address multiple issues together in a cohesive way. This planning is still in the early stages but we'll share more details when we have it. These real-world examples are very useful to inform this process!

[antgel]

@apparentlymart Great to hear, looking forward to updates here. Here's our very simple example, where the gw_aws_vpc creates a VPC and subnets...

# No alias => default provider
provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region     = "us-east-1"
}

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  **alias      = "ca-central-1"**
  region     = "ca-central-1"
}

module "vpc-us-east-1-sandbox" {
  source = "modules/gw_aws_vpc"
  environment = "sandbox"
  cidr_block = "10.10.0.0/16"
}

module "vpc-ca-central-1-production" {
  source = "modules/gw_aws_vpc"
  **provider = "aws.ca-central-1" # FAIL**
  environment = "production"
  cidr_block = "10.21.0.0/16"
}

[Techbrunch]

I think this issue is somewhat related to https://github.com/terraform-providers/terraform-provider-aws/issues/712, since we need to create some resources in a specific region no matter what is the region of the provider.

[fstuck37]

Hi All,
To add another use case. We have a lambda function for sending AWS VPC Flow logs to Splunk. The Lambda function, associated role, and permissions need to be deployed once in every region we require. I would like to deploy this in one TF script rather then one per region.

Not sure if this would help and probably over simplifying it but here's an idea I had:
module "mod_name" {
source = "...."
provider = "aws.alias"
}

Inside the module have some defined variable say module.provider that can per interpolated in the provider string. If no provider string is provided module.provider would be equal to the default provider.
resource "aws_instance" "test" {
provider = "${module.provider}"
...
}

Thanks,
Fred

[apparentlymart]

Hi all! It's been a while since I posted here so I just wanted to share an update.

Over the last couple months we've prototyped a few different approaches to this problem and we think we've settled on a good approach to move forward with. Providers are an unusual object in Terraform in two ways:

• They break the usual rule of modules being entirely encapsulated: to avoid a lot of verbosity, Terraform allows default (unaliased) providers to be defined in the root module and be automatically inherited by child modules, since we know that the common case is that users want to use the same provider settings throughout.
• Unlike most other blocks in configuration, provider config blocks are needed for _all_ operations on a resource: refreshing, updating, diffing, destroying, etc. This presents some interesting challenges when a child module containing its own provider blocks is removed from configuration, because this removes the provider configuration along with the rest, preventing Terraform from refreshing or destroying the now-removed resources. (This currently causes the counter-intuitive behavior discussed in #15778.)

With the above two constraints in mind, we decided to optimize for the approach of having all of the provider blocks in the root module but then being able to _pass_ providers down into child modules. provider blocks are still _allowed_ in descendant modules, but this will not be the recommended approach since it will still cause the issues discussed in #15778.

In the simple case with only default (unaliased) providers, the behavior will be the same as before: child modules can just implicitly use the provider configuration from their parent without any explicit declaration.

For more complex scenarios involving multiple instances of the _same_ provider, a new providers argument is supported inside a modules block which overrides that default inheritance behavior and instead explicitly passes certain providers to the child:

### DESIGN SKETCH: may change before release ###

# Default (unaliased) provider is used for most purposes
provider "aws" {
  region = "us-west-1"
}

# Alternative (aliased) aws provider instance is used for our child module
provider "aws" {
  region = "us-east-1"
  alias  = "use1"
}

module "example" {
  source = "./example"

  providers = {
    "aws" = "aws.use1"
  }
}

With the above configuration, any aws resources in module.example will use the configuration associated with aws.use1 in the root. With the providers argument set, module.example gets its own local "view" of the root provider configurations instead of implicitly inheriting the default providers.

This design also allows passing in _aliased_ providers to the child module, for situations where a module needs to work with more than one instance:

### DESIGN SKETCH: may change before release ###

provider "aws" {
  region = "us-west-1"
  alias  = "usw1"
}
provider "aws" {
  region = "us-east-1"
  alias  = "use1"
}

module "network_tunnel" {
  source = "./network-tunnel"

  providers = {
    "aws.src" = "aws.usw1"
    "aws.dst" = "aws.use1"
  }
}

With the provider blocks only in the _root_ module, we can safely remove the child module blocks without losing the provider configurations they use. Terraform will "remember" (in state) which provider config block (by reference) was last used for each resource so that it can use the same configuration for destruction if the module block is no longer available.

We're still working through some of the finer details of this, since it requires some re-organization of how Terraform deals with provider configurations internally, but we're hoping to be able to ship this new approach soon.

[rcrelia]

Great news, thank you for the update!

[jbardin]

Hi,

A minor clarification from the last update:

In order for modules to use a specific provider, they will need to have an unconfigured provider block to be named in the providers mapping as a sort of placeholder for concrete provider instance. This makes it clear that they can receive provider configuration, will allow us to more easily generate documentation, and allows modules to explicitly pass that same provider along to other submodules.

For example, the "network_tunnel" module above would have two blocks like so:

provider "aws" {
  alias  = "src"
}
provider "aws" {
  alias  = "dst"
}

[apparentlymart]

The new configuration feature from my earlier comment -- with the later modification noted by @jbardin -- is now merged into master and coming as part of Terraform 0.11.0.

This functionality is available in 0.11.0-beta1 for testing in experimental configurations. There is a bugfix around module destruction coming in 0.11.0-rc1 but otherwise we believe that the functionality in the beta should be complete. Please see the upgrade guide linked from the changelog for more information, along with the current documentation (which will appear on the website once we reach 0.11.0 final).

I'm going to close this issue now. If anyone tries the new functionality and find bugs or difficulties with it, please open a new top-level issue so that we can track each problem separately. This more-general issue will no longer be monitored.

Thanks for your patience here, everyone! Hopefully this new approach makes the interactions between providers and modules more intuitive and convenient.