Terraform: Multiple level module nesting fails

[bondsbw]

Terraform Version

Terraform v0.13.0-beta2

Terraform Configuration Files

Three files in this directory structure:

|   main.tf
\---a
    |   main.tf
    \---b
            main.tf

./main.tf:

module "a" {
  source = "./a"
}

./a/main.tf:

module "b" {
  source = "./b"
}

./a/b/main.tf:

data "external" "terraform" {
  program = ["terraform"]
}

Debug Output

https://gist.github.com/bondsbw/6207bf881f0f7230c4f16c462b89e474

Expected Behavior

Plan should have executed normally.

Actual Behavior

Error message:

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

Failed to instantiate provider "registry.terraform.io/hashicorp/external" to
obtain schema: unknown provider "registry.terraform.io/hashicorp/external"

Steps to Reproduce

Create file structure. Then run:

terraform init
terraform plan

Additional Context

Running on Windows 10 version 10.0.17763.0.

References

Related to #25282.

[bondsbw]

Note, this only occurs when the external provider is nested more than once. If nested only one time (for example, replacing the contents of ./a/main.tf with the contents of ./a/b/main.tf), the plan executes normally.

[bondsbw]

I also attempted to run terraform 0.13upgrade at all module levels. It produced a ./a/b/versions.tf file as follows, but did not fix the error.

terraform {
  required_providers {
    external = {
      source = "hashicorp/external"
    }
  }
  required_version = ">= 0.13"
}

[debovema]

Also experiencing this issue. A non-acceptable workaround is to duplicate the required_providers section in the root main.tf file it seems.

[bondsbw]

@debovema The workaround worked in my main project.

Is this a bug or expected behavior? I couldn't find the 0.13 documentation to verify, and terraform 0.13upgrade does not add requirements for deeper modules.

[debovema]

I think it's a bug due to the rework on modules to support depends_on and count.

[lorengordon]

Sure seems like a bug... even when the provider _is_ present, apply throws the error that it can't find it. Here's another repro setup, using TF_PLUGIN_CACHE_DIR to show the provider is present...

$ tree
.
├── foo
│   ├── bar
│   │   └── main.tf
│   └── main.tf
└── main.tf

$ cat main.tf
module "foo" {
  source = "./foo"
}

output "foo" {
  value = module.foo
}

$ cat foo/main.tf
module "bar" {
  source = "./bar"
}

output "bar" {
  value = module.bar
}

$ cat foo/bar/main.tf
resource "random_string" "this" {
  length  = 6
}

output "foo" {
  value = random_string.this.result
}

init

$ rm -rf .terraform/ terraform.tfstate*
$ ~/terraform013-beta2 init
Initializing modules...
- foo in foo
- foo.bar in foo/bar

Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

apply:

$ ~/terraform013-beta2 apply

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

Failed to instantiate provider "registry.terraform.io/hashicorp/random" to
obtain schema: unknown provider "registry.terraform.io/hashicorp/random"

but the provider is definitely present:

$ echo $TF_PLUGIN_CACHE_DIR
/home/loren/.terraform.d/plugin-cache
$ tree /home/loren/.terraform.d/plugin-cache/registry.terraform.io/
/home/loren/.terraform.d/plugin-cache/registry.terraform.io/
└── hashicorp
    ├── aws
    │   └── 2.66.0
    │       └── linux_amd64
    │           └── terraform-provider-aws_v2.66.0_x4
    ├── external
    │   └── 1.2.0
    │       └── linux_amd64
    │           └── terraform-provider-external_v1.2.0_x4
    ├── null
    │   └── 2.1.2
    │       └── linux_amd64
    │           └── terraform-provider-null_v2.1.2_x4
    └── random
        └── 2.2.1
            └── linux_amd64
                └── terraform-provider-random_v2.2.1_x4

and note that it does work when using beta1...

$ rm -rf .terraform/
$ ~/terraform013-beta1 init
Initializing modules...
- foo in foo
- foo.bar in foo/bar

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/random...
- Using hashicorp/random v2.2.1 from the shared cache directory

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/random: version = "~> 2.2.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ ~/terraform013-beta1 apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.foo.module.bar.random_string.this will be created
  + resource "random_string" "this" {
      + id          = (known after apply)
      + length      = 6
      + lower       = true
      + min_lower   = 0
      + min_numeric = 0
      + min_special = 0
      + min_upper   = 0
      + number      = true
      + result      = (known after apply)
      + special     = true
      + upper       = true
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.foo.module.bar.random_string.this: Creating...
module.foo.module.bar.random_string.this: Creation complete after 0s [id=hXPB_(]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

foo = {
  "bar" = {
    "foo" = "hXPB_("
  }
}

[lorengordon]

Note differences in the .terraform contents between beta1 and beta2, in particular .terraform/plugins/selections.json...

using beta2:

$ rm -rf .terraform/ terraform.tfstate*
$ ~/terraform013-beta2 init
Initializing modules...
- foo in foo
- foo.bar in foo/bar

Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ tree .terraform/
.terraform/
├── modules
│   └── modules.json
└── plugins
    └── selections.json

2 directories, 2 files

$ cat .terraform/modules/modules.json
{"Modules":[{"Key":"foo","Source":"./foo","Dir":"foo"},{"Key":"foo.bar","Source":"./bar","Dir":"foo/bar"},{"Key":"","Source":"","Dir":"."}]} 

$ cat .terraform/plugins/selections.json
{}

using beta1:

$ rm -rf .terraform/ terraform.tfstate*
$ ~/terraform013-beta1 init
Initializing modules...
- foo in foo
- foo.bar in foo/bar

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/random...
- Using hashicorp/random v2.2.1 from the shared cache directory

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/random: version = "~> 2.2.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ tree .terraform/
.terraform/
├── modules
│   └── modules.json
└── plugins
    ├── registry.terraform.io
    │   └── hashicorp
    │       └── random
    │           └── 2.2.1
    │               └── linux_amd64 -> /home/loren/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/random/2.2.1/linux_amd64
    └── selections.json

7 directories, 2 files

$ cat .terraform/modules/modules.json
{"Modules":[{"Key":"","Source":"","Dir":"."},{"Key":"foo","Source":"./foo","Dir":"foo"},{"Key":"foo.bar","Source":"./bar","Dir":"foo/bar"}]}

$ cat .terraform/plugins/selections.json
{
  "registry.terraform.io/hashicorp/random": {
    "hash": "h1:Zg1Bpi6vr7b0H6no8kVDfEucn5pvNALivdrVKVHarGs=",
    "version": "2.2.1"
  }
}

[alisdair]

Thanks to everyone who contributed to this issue for the very detailed investigation into this, and the reduced reproduction case! This bug has been fixed in #25334 and will be included in the upcoming 0.13.0 prerelease.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.