Terraform: Terraform 0.13 referencing nonexistent HashiCorp provider when init

[vulpine]

Using a local provider, Terraform 0.13 correctly discovers it after terraform state replace-provider, but still fails to find a phantom hashicorp/twilio which we do not appear to have referenced anywhere.

We previously had a provider named twilio which we had in ~/.terraform.d/plugins/terraform-provider-twilio, and we have moved it into a local directory structure using a virtual namespace as ~/.terraform.d/plugins/terraform.yelp.com/yelp/twilio/1.0.0/darwin_amd64/terraform-provider-twilio_v1.0.0. We've amended our module to refer to this new virtual namespace and version.

terraform state replace-provider 'registry.terraform.io/-/twilio' 'terraform.yelp.com/yelp/twilio' made the following changes throughout our state file:

-      "provider": "provider.twilio",
+      "provider": "provider[\"terraform.yelp.com/yelp/twilio\"]",

However, terraform init fails per the linked Gist. It still seems that somewhere a hashicorp/twilio provider is being referenced:

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/twilio]
└── module.base_config
    └── provider[terraform.yelp.com/yelp/twilio] 1.0.0

Providers required by state:

    provider[terraform.yelp.com/yelp/twilio]

We can't see any reference to hashicorp/twilio in our codebase: it's not referenced in state, and this same problem occurs even if we remove our local state and try to init from scratch.

Terraform Version

Terraform v0.13.0
+ provider terraform.yelp.com/yelp/twilio v1.0.0

Terraform Configuration Files

terraform {
  required_providers {
    twilio = {
      source  = "terraform.yelp.com/yelp/twilio"
      version = "1.0.0"
    }
  }
}

provider "twilio" {
  allowed_account_sid = "sometextgoesherebutiblankeditout"
}

Debug Output

https://gist.github.com/vulpine/6bb4020bdec14c8de162d82014965214

Crash Output

Expected Behavior

terraform init should have initialised correctly with our local module. terraform providers should not show a registry.terraform.io/hashicorp/twilio provider.

Actual Behavior

terraform init fails due to a lack of registry.terraform.io/hashicorp/twilio:

Error while installing hashicorp/twilio: provider registry
registry.terraform.io does not have a provider named
registry.terraform.io/hashicorp/twilio

Steps to Reproduce

terraform init

Additional Context

Terraform 0.13 from Homebrew on macOS. No previous attempts to init this were made locally but the state file had been migrated from 0.12. The issue persists even with the state removed.

References

[alisdair]

Thanks for reporting this issue!

From the terraform providers output, it looks like your root module has a dependency on an unspecified twilio provider, which Terraform is interpreting as hashicorp/twilio. It's hard to say how this is happening without seeing the configuration. Do you have a provider "twilio" {} block or any resource "twilio_[…]" blocks in the root module?

All modules using a non-default provider need a required_providers block specifying the provider's source address. Perhaps you could try adding a required_providers block at the root module.

If that explanation doesn't help, can you try creating a minimal Terraform configuration which demonstrates the issue? That would help us reproduce and fix any bug this uncovers. Thanks again!

[vulpine]

@alisdair Thanks for the prompt reply!

The only place we had a provider "twilio" {} at time of raising this issue was in our sourced module, ../../modules/twilio_config. Previously, this had been in the root module, in a providers.tf file.

# Previous providers.tf file - no longer extant.
provider "twilio" {
    allowed_account_sid = "sometextgoesherebutiblankeditout"
}

I had a required_providers block in my sourced module; if I move that into my root module, into that same providers.tf, the situation doesn't change:

# With a `providers.tf` file in my root, but not in the module
% cat base_config.tf
module "base_config" {
  source = "../../modules/twilio_config"
}
% ls -l $_
-rw-r--r--  1 smatthew  1954207199  219 25 Aug 22:38 ../../modules/twilio_config/providers.tf
% mv ../../modules/twilio_config/providers.tf ./providers.tf
% cat providers.tf
terraform {
  required_providers {
    twilio = {
      source  = "terraform.yelp.com/yelp/twilio"
      version = "1.0.0"
    }
  }
}
provider "twilio" {
  allowed_account_sid = "sometextgoesherebutiblankeditout"
}
% terraform providers
Providers required by configuration:
.
├── provider[terraform.yelp.com/yelp/twilio] 1.0.0
└── module.base_config
    └── provider[registry.terraform.io/hashicorp/twilio]

Providers required by state:

    provider[terraform.yelp.com/yelp/twilio]

And if I copy that back into the sourced module, so it's in both, everything works perfectly!

# With an identical `providers.tf` file in my root and in my module
% cp providers.tf ../../modules/twilio_config/providers.tf
% terraform providers
Providers required by configuration:
.
├── provider[terraform.yelp.com/yelp/twilio] 1.0.0
└── module.base_config
    └── provider[terraform.yelp.com/yelp/twilio] 1.0.0

Providers required by state:

    provider[terraform.yelp.com/yelp/twilio]

And, I can even init! :) So we've a workaround now.

Does this demonstrate the issue sufficiently, or would you like some more code fragments / config to help reproduce?

[alisdair]

Ah, interesting!

Your fix of having the required_providers block in both the root module and the base_config child module would make sense, if both modules refer to the provider somehow. Is that the case?

If it's not, and there are no references to twilio in the root module, then this indeed sounds like a bug. Sharing as much of the code of the root module as possible (or providing a reduced reproduction) would be very helpful.

All modules which use a provider with a non-default source need a required_providers block so that Terraform knows how to map the local name (e.g. twilio) to the full provider source address (e.g. terraform.yelp.com/yelp/twilio). It's this last full address that Terraform uses to map resources to providers.

[dpkirchner]

What do you think about adding a way to supply non-default sourced providers to the child modules from the root module? I'm picturing something that works like provider aliases. This would allow us to use locally modified providers with off-the-shelf modules.

[ivantomica]

Using 0.13.1 here and facing what appears to be the same issue. Previously I've referenced provider "hcloud" only, but now I have following in my main.tf:

    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "1.20.1"
    }

Yet still, during the plan terraform is trying to fetch non-existent provider in addition to the correct one:

Initializing provider plugins...
- Finding hetznercloud/hcloud versions matching "1.20.1"...
2020/08/31 20:21:13 [DEBUG] GET https://registry.terraform.io/v1/providers/hetznercloud/hcloud/versions
2020/08/31 20:21:13 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hetznercloud/hcloud/versions
- Finding latest version of hashicorp/hcloud...
2020/08/31 20:21:13 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/hcloud/versions
2020/08/31 20:21:13 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/hcloud/versions

From what I see I don't have any explicit dependencies on hashicorp/hcloud, or hcloud only if you wish. I've also issued replace-provider and replaced it on all existing resources in the state file.

Setup using hcloud resources is configured in a way that I have servers.tf file which each includes module "NAMEOFTHESERVER" and sources external repository.

Once I comment out whole servers.tf, init goes through normally. So it would appear that something within the called module is trying to fetch hcloud module. If on the other hand I switch the source of the module to local path (so I'm sure changes I'm making are being sourced without committing to the repository) and comment out all "hcloud_*" related resources, it still tries to fetch hashicorp/hcloud provider. So not really sure from where does the requirement come from.

If I can gather any additional info, please let me know.

EDIT:

What I failed to do is:
https://www.terraform.io/upgrade-guides/0-13.html

Each module must declare its own set of provider requirements, so if you have a configuration which calls other modules then you'll need to run this upgrade command for each module separately.

Once I did that, all started working fine. Why I missed this thing is the fact that I have both modules and main code in the same repository, and in order to upgrade the module, you have to CD to that module directory and issue

terraform 0.13upgrade

from there.

[ctrox]

I just ran into the same thing. Does this mean every child module has to maintain the provider version? That makes upgrading providers accross lots of modules really annoying, since we have to maintain the version in like 100 different modules. Or is there a way to just force a version from the root module?

[alisdair]

I just ran into the same thing. Does this mean every child module has to maintain the provider version?

No. Provider version arguments across multiple modules in a configuration are merged, so having a single version constraint at the root module is fine.

What is necessary in all modules is an explicit source argument, for providers which are not default (i.e. HashiCorp providers hosted on the public registry). This is necessary for Terraform to determine which fully-qualified provider name maps to a given local name for each module.

Said another way:

• Provider version constraints can be specified once, at the root module, if that makes sense for your modules
• Provider source can be omitted for HashiCorp providers
• For other providers, source must be specified in each module using the provider

I hope that makes sense! For more details see the docs on provider requirements and the 0.13 upgrade guide. If you have any further questions, please start a topic in the Terraform discussion forum.

[ivantomica]

Nice explanation @alisdair , thank you!

One question though, what's the blocker of having provider source set only on root module and propagated to all sub-modules, unless specified differently? Such behaviour would feel consistent.

[alisdair]

One question though, what's the blocker of having provider source set only on root module and propagated to all sub-modules, unless specified differently?

Good question! It's a philosophical choice. For modules to be considered reusable chunks of Terraform, they ought to define their own dependencies. Just like all modules must declare their input variables, we believe that modules should declare the providers that they depend on.

A module relying on the implicit configuration should mean "use the HashiCorp provider", not "use the HashiCorp provider, unless the caller overrides". The second of these would be inconsistent with the principle of modules having a consistent defined interface to their callers.

[ivantomica]

For modules to be considered reusable chunks of Terraform, they ought to define their own dependencies. Just like all modules must declare their input variables, we believe that modules should declare the providers that they depend on.

On that note, shouldn't the version for the very same provider, be defined on the module level as well? This way it seems to me that there's possibility for conflicts when you have provider specified at two places. Namely, when defining version on the root module vs having different version on the sub-module level.

Anyhow, I now consider myself well-informed on the issue and at least know how to troubleshoot such issues in the future :-) I really hope though you reconsider this provider part to behave something like:

• If provider is defined on the root level, but not on the module level, use that info.
• If not defined on the root level, error out.
• If provider defined on both root level and module level, use module layer configuration.

This way, it would still be true that you're encouraged to specify provider source at the module level, but if not specified, well, at least provide definition globally. To some extent, same reasoning could be applied to the version, which can only be specified once though.

[randomswdev]

@alisdair I have a large set of Terrafrom scripts that, because of the large number of resources it contains, is organized into folders. Each folder (i.e. module) contains resources related to a specific domain.

I have about 200 folders and Terraform version 0.12 successfully processes the submodules, which include hashicorp providers, thnird party providers and home grown providers. Upgrading to version 0.13 requires me to add provider constraints to about 200 "modules", each one with its own requirements, which is absolutely crazy. I know that modules where not intended to be used as I'm using them, but I think many people are using them my way.

I understand that Terrafrom is version 0.x so not yet "GA", but as a matter of fact lots of people use it (and you sell it), so backward compatibility is mandatory at this point. Do you have any plan to allow submodules inherit the source of providers as the version is inherited, as @ivantomica was suggesting?

[alisdair]

Do you have any plan to allow submodules inherit the source of providers as the version is inherited, as @ivantomica was suggesting?

No, there are no plans to do this. As I mentioned above, our position is that that modules should be independent reusable blocks of configuration, which therefore must specify their dependencies. As of 0.13, with the introduction of provider source and third-party providers on the Terraform Registry, this means defining the source for non-HashiCorp providers.

I'm sorry that this isn't the solution you're looking for. As maintainers of Terraform, we have to make judgement calls at times about the best compromise for a given situation, and we unfortunately can't always make everyone's workflow equally easy.

For your specific situation, I can suggestion a possible workaround to make the migration process easier. If all of your "modules" have the same provider requirements, you may want to define a project-wide providers.tf file and symlink it into the individual directories. Even if some modules don't use all of the providers, this will be valid. I hope that is a reasonable compromise!

This discussion has diverged quite far from the original bug report, which it seems has already been resolved, so I'm going to close this issue now.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.