Terraform: "provider configuration block is required for all operations" when module not used

[nodesocket]

I have a module api that dynamically takes in a variable:

variable "provider" {
    type = "map"
    description = "The provider map to use for the module"
}

In that api module the provider is then defined:

provider "aws" {
    profile = "${var.provider["profile"]}"
    shared_credentials_file = "${var.provider["shared_credentials_file"]}"
    region = "${var.provider["region"]}"
}

Currently I am NOT instantiating a single instance of the api module (commented out), but when running apply or plan erroring with:

configuration for module.api_eu_central_1.provider.aws is not present; a provider configuration block is required for all operations

I confirmed that the api module instance is indeed commented out, why is Terraform looking for the provider block at all if no instance of the module is defined?

[jbardin]

Hi @nodesocket,

This should only happen if there is still a resource in the state using the module.api_eu_central_1.provider.aws provider. If there are resources in the module that you intend to destroy by removing the module, they will still need an appropriately configured provider in order to be destroyed.

We recommend configuring the providers at the top level of the configuration, and passing them into the modules so that modules can be modified and removed independently from the providers.

[nodesocket]

@jbardin I have confirmed that there are no instances the module api_us_west_2 that are active, but still running plan or apply errors with:

Error: Error asking for user input: 1 error(s) occurred:

* module.api_us_west_2.data.aws_ami.api: configuration for module.api_us_west_2.provider.aws is not present; a provider configuration block is required for all operations

Are data sources invoked even if the module in which they are defined are not used?

[jbardin]

Ah ha, that must be it! Data sources are a type of resource, and they are persisted in the state. Removing a module would try to connect them to a provider which might no longer exist.

Technically destroying a data source shouldn't need a provider, but in practice if there is a data source in a module it's usually because there is some accompanying resource that depends on it, and there's a chance the data source may need to be refreshed in order to correctly destroy the module. Following the prescribed practice of explicitly passing providers into modules will avoid these issues entirely

Also note that even of the resources are no longer around, if they exist in terraform's state, terraform doesn't know they no longer exist and will need the provider to verify that.

We might be able to detect and skip the destruction of an unused data source when removing a module, so I'll leave this open as a possible enhancement.

[nodesocket]

@jbardin I really appreciate the help.

Following the prescribed practice of explicitly passing providers into modules will avoid these issues entirely.

Sorry I am not clear, but I think I am currently doing this:

    modules
         api
              provider.tf
              variables.tf
    environments
         production
               main.tf
               provider.tf
````

*environments/production/provider.tf*

provider "aws" {
shared_credentials_file = "aws-auth.ini"
profile = "default"
region = "us-west-2"
}

*environments/production/main.tf*

module "api_us_west_2" {
source = "../../modules/api"
provider = {
profile = "default"
shared_credentials_file = "aws-auth.ini"
region = "us-west-2"
}
...
}

*modules/api/variables.tf*

variable "provider" {
type = "map"
description = "The provider map to use for the module"
}

*modules/api/provider.tf*

provider "aws" {
profile = "${var.provider["profile"]}"
shared_credentials_file = "${var.provider["shared_credentials_file"]}"
region = "${var.provider["region"]}"
}
```

[jbardin]

You have a provider defined in modules/api, so removing that module will remove the provider configuration, which means any resources that need to be destroyed won't have a provider configured to do so. If you're ever going to remove a module to destroy the resources, you need to define the provider _outside_ the module so that it still exists for the destruction process. See Providers Within Modules

[nodesocket]

Getting really close now, but still getting an error:

Error: module "api_eu_central_1": cannot pass non-existent provider "euc1"

Here is the new setup:

environments/production/providers.tf

provider "aws" {
    shared_credentials_file = "aws-auth.ini"
    profile = "default"
    region = "us-west-2"
}

provider "aws" {
    alias = "euc1"
    region = "eu-central-1"
}

environments/production/main.tf

module "api_us_west_2" {
    source = "../../modules/api"
    ....
}

module "api_eu_central_1" {
    source = "../../modules/api"
    providers = {
        aws = "euc1"
    }
    ...
}

Deleted the api module provider.tf and provider map variable inside the api module.

[jbardin]

The provider is named "aws.euc1"

providers = {
    aws = "aws.euc1"
}

[nodesocket]

Ok, perfect that worked with modules both instantiated. When I commend them both out _(no instances of the modules)_ back to this error:

> Error: Error asking for user input: 1 error(s) occurred:

* module.api_us_west_2.aws_lb_target_group_attachment.api1_public_alb_tga_80: configuration for module.api_us_west_2.provider.aws is not present; a provider configuration block is required for all operations

[jbardin]

configuration for module.api_us_west_2.provider.aws is not present

This looks like you still have an aws provider in the module configuration. Have you removed that provider config?

[nodesocket]

This is what is so strange. I just doubled checked and no provider {} block inside the api module at all.

[nodesocket]

@jbardin could this be related to using a count property in a resource inside of the API module?

The error is:

Error: Error asking for user input: 1 error(s) occurred:

* module.api_us_west_2.aws_lb.api_public_alb: configuration for module.api_us_west_2.provider.aws is not present; a provider configuration block is required for all operations

Looking at that resource it is defined as:

resource "aws_lb" "api_public_alb" {
    count = "${var.primary_region}"
    name = "${var.name}public"
    load_balancer_type = "application"
    subnets = [
        "${var.subnet_ids}"
    ]
    security_groups = [
        "${var.alb_public_security_groups}"
    ]
    enable_http2 = true
    ip_address_type = "ipv4"
}

[jbardin]

No, the count shouldn't effect it at all (though I'm not at all sure what is going on yet).

The module.api_us_west_2.aws_lb.api_public_alb resource is referencing module.api_us_west_2.provider.aws, which means that an aws provider is being defined in that module. Implicit providers are only created within the root module's namespace, so that would suggest there is something defining that provider in the api_us_west_2 module.

[nodesocket]

I just triple checked and even did a quick grep of the entire api module directory for provider and have zero results.

I've noticed sometimes terraform get's out of sync and I have to delete the .terraform directory locally and do terraform init. I've tried this multiple times but the same behavior. I am using a Google Storage bucket for remote storage, and I hope that the remote storage state file did not get corrupted somehow.

I'll post the entire commented out instances of the api module in question. Maybe there is something I am missing. Note, I am not defining a provider for api_us_west_2 as that should be the default.

# module "api_us_west_2" {
#     source = "../../modules/api"
#     name = "api"
#     ssh_private_key = "**REDACTED**"
#     vpc_id = "**REDACTED**"
#     subnet_ids = [
#         "**REDACTED**", //us-west-2a
#         "**REDACTED**", //us-west-2b
#         "**REDACTED**"  ///us-west-2c
#     ]
#     alb_public_security_groups = "${module.security_groups_us_west_2.api_public_alb_security_group}"
#     alb_private_security_groups = "${
#         list(
#             module.security_groups_us_west_2.api_private_alb_security_group
#         )
#     }"
#     security_groups = "${
#         list(
#             module.security_groups_us_west_2.base_security_group,
#             module.security_groups_us_west_2.api_security_group
#         )
#     }"
#     dns_zone_id = "**REDACTED**"
#     certificate_arn = "**REDACTED**"
#     api_dns_alias = "**REDACTED**"
#     logs_dns_alias = "**REDACTED**"
#     release_dns_alias = "**REDACTED**"
#     iam_instance_profile = "${module.iam.secrets_profile}"
#     secrets_bucket = "**REDACTED**"
#     instance_type = "t2.small"
#     ebs_optimized = false
# }

# module "api_eu_central_1" {
#     source = "../../modules/api"
#     providers = {
#         aws = "aws.euc1"
#     }
#     primary_region = false
#     name = "api"
#     ssh_private_key = "**REDACTED**"
#     vpc_id = "**REDACTED**"
#     subnet_ids = [
#         "**REDACTED**", //eu-cenetral-1a
#         "**REDACTED**", //eu-cenetral-1b
#         "**REDACTED**"  //eu-cenetral-1c
#     ]
#     alb_private_security_groups = "${
#         list(
#             module.security_groups_eu_central_1.api_private_alb_security_group
#         )
#     }"
#     security_groups = "${
#         list(
#             module.security_groups_eu_central_1.base_security_group,
#             module.security_groups_eu_central_1.api_security_group
#         )
#     }"
#     api_dns_alias = "**REDACTED**"
#     logs_dns_alias = "**REDACTED**"
#     release_dns_alias = "**REDACTED**"
#     iam_instance_profile = "${module.iam.secrets_profile}"
#     secrets_bucket = "**REDACTED**"
#     instance_type = "t2.small"
#     ebs_optimized = false
# }

[jbardin]

Sorry, I assumed you were starting from scratch each time to reproduce the issue. If the reference to module.api_us_west_2.provider.aws is already stored in the state, then you need to put that version of the module back in order to apply any changes. I think we need to back up here and take it step by step.

The order of operations here is:

• Start with the original module configuration, containing the provider block. A plan here should show no changes.
• Move the provider definitions outside of the module. A plan should still show no changes, and running apply will update provider references in the state.
• Once the state is updated to match the new configuration, then the module can be removed, destroying any remaining resources in contained.

If you're still at an impasse, the trace logs would be helpful to show what's actually going on.

[jbardin]

I missed the last update to your previous comment, but you also only have a providers block in the api_eu_central_1 definition, not in the api_us_west_2 which is using the same source.

[nodesocket]

Ok so magically running apply with both module instances active _(not commented out)_ even though no changes were applied fixed it. When I commented both module instances out, now it is working and showing all destroy operations?

Is that expected behavior that apply is required even though no resources changed?

but you also only have a providers block in the api_eu_central_1 definition, not in the api_us_west_2 which is using the same source.

Correct, api_us_west_2 is using the default provider and api_eu_central_1 is using the aliased.

provider "aws" {
    shared_credentials_file = "aws-auth.ini"
    profile = "default"
    region = "us-west-2"
}

provider "aws" {
    alias = "euc1"
    shared_credentials_file = "aws-auth.ini"
    profile = "default"
    region = "eu-central-1"
}

[jbardin]

Yes, the apply was required to update the changed location of the provider in the state, so that when the module was removed the resource could be destroyed with no config present.

[nodesocket]

When in doubt, apply seems to be a good rule then. I was running plan, and never applied because it showed no changes.

[nodesocket]

I am going to close this. Thanks so much for the help @jbardin.

[volker-fr]

Just came across this issue and after the first two workarounds didn't work well for me I would propose a third workaround:

Change

module "api_us_west_2" {
   source = "../../modules/api"
   ...
}

to

module "api_us_west_2" {
  source = "./temp-api_us_west_2-module-for-deletion"
   ...
}

copy the existing module to ./temp-api_us_west_2-module-for-deletion and then remove all the files that create resources etc (put not the provider). Then run the usual terraform plan && teraform apply

[bradenwright]

I had multi levels of modules so the last comment about copying folders didn't work for me. It was really easy to just edit the statefile (of course be careful). I found:

"provider": "module.sandbox2.module.dns-external.provider.ns1"
in a few places and simply had to change it to:

"provider": "provider.ns1"

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.