Terraform: Error: Failed to install provider for thirdparty provider

[cpesch]

Terraform Version

0.13.0 beta1

Terraform Configuration Files

provider "keycloak" {
  url = "https://XXX"
  client_id = "XXX"
}

terraform {
  required_providers {
    keycloak = {
      source  = "github.com/mrparker/keycloak"
      version = "1.19.0"
    }
  }
  required_version = ">= 0.13"
}

module "XXX-openid-client" {
  source = "git::https://XXX/tfmodule-keycloak-client"

  for_each = local.realms
  realm_id = keycloak_realm.customer_realm[each.key].realm
  base_urls = [
    "https://${each.key}-XXX"
  ]
}

resource "keycloak_openid_client" "XXX" {
  realm_id  = var.realm_id
...
  client_id = "XXX"
}

Debug Output

$ terraform init
2020/06/11 20:23:10 [INFO] Terraform version: 0.13.0 beta1 
2020/06/11 20:23:10 [INFO] Go runtime version: go1.14.4
2020/06/11 20:23:10 [INFO] CLI args: []string{"/Users/cpesch/go/src/github.com/hashicorp/terraform/tfmaster", "init"}
2020/06/11 20:23:10 [DEBUG] Attempting to open CLI config file: /Users/cpesch/.terraformrc
2020/06/11 20:23:10 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/06/11 20:23:10 [DEBUG] checking for credentials in "/Users/cpesch/.terraform.d/plugins"
2020/06/11 20:23:10 [DEBUG] checking for credentials in "/Users/cpesch/.terraform.d/plugins/darwin_amd64"
2020/06/11 20:23:10 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/06/11 20:23:10 [DEBUG] will search for provider plugins in /Users/cpesch/.terraform.d/plugins
2020/06/11 20:23:10 [TRACE] getproviders.SearchLocalDirectory: found github.com/mrparker/keycloak v1.18.0 for darwin_amd64 at /Users/cpesch/.terraform.d/plugins/github.com/mrparker/keycloak/1.18.0/darwin_amd64
2020/06/11 20:23:10 [TRACE] getproviders.SearchLocalDirectory: found github.com/mrparker/keycloak v1.19.0 for darwin_amd64 at /Users/cpesch/.terraform.d/plugins/github.com/mrparker/keycloak/1.19.0/darwin_amd64
2020/06/11 20:23:10 [DEBUG] ignoring non-existing provider search directory /Users/cpesch/Library/Application Support/io.terraform/plugins
2020/06/11 20:23:10 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2020/06/11 20:23:10 [INFO] CLI command args: []string{"init"}
2020/06/11 20:23:10 [TRACE] ModuleInstaller: installing child modules for . into .terraform/modules
Initializing modules...
2020/06/11 20:23:10 [DEBUG] Module installer: begin labs-linkfinder-openid-client
2020/06/11 20:23:10 [TRACE] ModuleInstaller: Module installer: labs-linkfinder-openid-client <nil> already installed in .terraform/modules/labs-linkfinder-openid-client
2020/06/11 20:23:10 [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json

2020/06/11 20:23:10 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
Initializing the backend...
2020/06/11 20:23:10 [TRACE] Preserving existing state lineage "0118919b-bc31-eb5c-b117-f40cf70f8c17"
2020/06/11 20:23:10 [TRACE] Preserving existing state lineage "0118919b-bc31-eb5c-b117-f40cf70f8c17"
2020/06/11 20:23:10 [TRACE] Meta.Backend: working directory was previously initialized but has no backend (is using legacy remote state?)
2020/06/11 20:23:10 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2020/06/11 20:23:10 [TRACE] Meta.Backend: instantiated backend of type <nil>
2020/06/11 20:23:10 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
2020/06/11 20:23:10 [TRACE] getproviders.SearchLocalDirectory: found github.com/mrparker/keycloak v1.19.0 for darwin_amd64 at .terraform/plugins/github.com/mrparker/keycloak/1.19.0/darwin_amd64
2020/06/11 20:23:10 [TRACE] providercache.fillMetaCache: including .terraform/plugins/github.com/mrparker/keycloak/1.19.0/darwin_amd64 as a candidate package for github.com/mrparker/keycloak 1.19.0
2020/06/11 20:23:10 [DEBUG] checking for provisioner in "."
2020/06/11 20:23:10 [DEBUG] checking for provisioner in "/Users/cpesch/go/src/github.com/hashicorp/terraform"
2020/06/11 20:23:10 [DEBUG] checking for provisioner in ".terraform/plugins/darwin_amd64"
2020/06/11 20:23:10 [DEBUG] checking for provisioner in "/Users/cpesch/.terraform.d/plugins"
2020/06/11 20:23:10 [DEBUG] checking for provisioner in "/Users/cpesch/.terraform.d/plugins/darwin_amd64"
2020/06/11 20:23:10 [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2020/06/11 20:23:10 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2020/06/11 20:23:10 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2020/06/11 20:23:10 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2020/06/11 20:23:10 [TRACE] statemgr.Filesystem: read nil snapshot

2020/06/11 20:23:10 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
Initializing provider plugins...
2020/06/11 20:23:10 [TRACE] getproviders.SearchLocalDirectory: found github.com/mrparker/keycloak v1.19.0 for darwin_amd64 at .terraform/plugins/github.com/mrparker/keycloak/1.19.0/darwin_amd64
2020/06/11 20:23:10 [TRACE] providercache.fillMetaCache: including .terraform/plugins/github.com/mrparker/keycloak/1.19.0/darwin_amd64 as a candidate package for github.com/mrparker/keycloak 1.19.0
2020/06/11 20:23:10 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2020/06/11 20:23:10 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
- Using previously-installed github.com/mrparker/keycloak v1.19.0
- Finding latest version of hashicorp/keycloak...
2020/06/11 20:23:11 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/keycloak/versions
2020/06/11 20:23:11 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/keycloak/versions
2020/06/11 20:23:11 [TRACE] providercache.fillMetaCache: using cached result from previous scan of .terraform/plugins
2020/06/11 20:23:11 [DEBUG] GET https://registry.terraform.io/v1/providers/-/keycloak/versions
2020/06/11 20:23:11 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/-/keycloak/versions

Error: Failed to install provider

Error while installing hashicorp/keycloak: provider registry
registry.terraform.io does not have a provider named
registry.terraform.io/hashicorp/keycloak

Expected Behavior

terraform init succeeds and allows me to test modules.for_each

Actual Behavior

Error: Failed to install provider

Error while installing hashicorp/keycloak: provider registry
registry.terraform.io does not have a provider named
registry.terraform.io/hashicorp/keycloak

although I'm using a local provider

Steps to Reproduce

1. terraform init

References

Followup of https://github.com/hashicorp/terraform/issues/25172

[alisdair]

It looks like the issue here is that the module "XXX-openid-client" has not been upgraded to use the new provider source feature. For non-HashiCorp providers, all modules in a configuration must define an explicit source. If a module uses a provider without an explicit source, it will default to the hashicorp namespace.

This is because we now allow different modules to use different providers with the same name. That means that "acme/dns" and "bluecorp/dns" can both be used as dns in different modules.

To fix this issue, you should add the same required_providers block to your client module so that it knows that keycloak is found at "github.com/mrparker/keycloak". Please let us know if that resolves the problem!

[cpesch]

@alisdair You're right. I forgot the -upgrade when doing terraform init.

With 0.13 beta1 it didn't work but with the master from yesterday I could finally use modules.for_each and it works :-).

[wolviecb]

I've seem to be stuck in the same error here with a local provider I've built. When I try to init with terraform 0.13 I get a Error: Failed to query available provider packages

Terraform Version

0.13.0 beta2

Terraform Configuration Files

terraform {
  required_version = ">= 0.13"
  required_providers {
    tranzaxis = {
      source  = "local/txr/tranzaxis"
      version = "0.0.1"
    }
  }
}


resource "tranzaxis_instance_state" "instance_1" {
  [...]
}

Debug Output

$ terraform init -plugin-dir=terraform.d
2020/06/23 12:57:05 [INFO] Terraform version: 0.13.0 beta2 
2020/06/23 12:57:05 [INFO] Go runtime version: go1.14.2
2020/06/23 12:57:05 [INFO] CLI args: []string{"/Users/thomas.andrade/txr/repos/tft/terraform-provider-tranzaxis/test/terraform", "init", "-plugin-dir=terraform.d"}
2020/06/23 12:57:05 [DEBUG] Attempting to open CLI config file: /Users/thomas.andrade/.terraformrc
2020/06/23 12:57:05 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/06/23 12:57:05 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/06/23 12:57:05 [DEBUG] ignoring non-existing provider search directory /Users/thomas.andrade/.terraform.d/plugins
2020/06/23 12:57:05 [DEBUG] ignoring non-existing provider search directory /Users/thomas.andrade/Library/Application Support/io.terraform/plugins
2020/06/23 12:57:05 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2020/06/23 12:57:05 [INFO] CLI command args: []string{"init", "-plugin-dir=terraform.d"}

Initializing the backend...
2020/06/23 12:57:05 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2020/06/23 12:57:05 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2020/06/23 12:57:05 [DEBUG] New state was assigned lineage "c4ad9d37-7a52-c2da-816b-bb1dd98483c9"
2020/06/23 12:57:05 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2020/06/23 12:57:05 [TRACE] Meta.Backend: instantiated backend of type <nil>
2020/06/23 12:57:05 [DEBUG] checking for provisioner in "terraform.d"
2020/06/23 12:57:05 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2020/06/23 12:57:05 [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2020/06/23 12:57:05 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup

Initializing provider plugins...
2020/06/23 12:57:05 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
- Finding local/txr/tranzaxis versions matching "0.0.1"...
2020/06/23 12:57:05 [TRACE] statemgr.Filesystem: read snapshot with lineage "f891f123-285a-6782-bf96-67c90c011f40" serial 10
2020/06/23 12:57:05 [DEBUG] init: overriding provider plugin search paths
2020/06/23 12:57:05 [DEBUG] will search for provider plugins in [terraform.d]
2020/06/23 12:57:05 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
2020/06/23 12:57:05 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/plugins: cannot search .terraform/plugins: lstat .terraform/plugins: no such file or directory
2020/06/23 12:57:05 [WARN] Failed to scan provider cache directory .terraform/plugins: cannot search .terraform/plugins: lstat .terraform/plugins: no such file or directory
2020/06/23 12:57:05 [TRACE] getproviders.SearchLocalDirectory: found local/txr/tranzaxis v0.0.1 for darwin_amd64 at terraform.d/local/txr/tranzaxis/0.0.1/darwin_amd64
2020/06/23 12:57:05 [TRACE] providercache.Dir.InstallPackage: installing local/txr/tranzaxis v0.0.1 from terraform.d/local/txr/tranzaxis/0.0.1/darwin_amd64
2020/06/23 12:57:05 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
2020/06/23 12:57:05 [TRACE] getproviders.SearchLocalDirectory: found local/txr/tranzaxis v0.0.1 for darwin_amd64 at .terraform/plugins/local/txr/tranzaxis/0.0.1/darwin_amd64
2020/06/23 12:57:05 [TRACE] providercache.fillMetaCache: including .terraform/plugins/local/txr/tranzaxis/0.0.1/darwin_amd64 as a candidate package for local/txr/tranzaxis 0.0.1
2020/06/23 12:57:05 [TRACE] providercache.fillMetaCache: using cached result from previous scan of .terraform/plugins
- Finding latest version of -/tranzaxis...
- Installing local/txr/tranzaxis v0.0.1...
- Installed local/txr/tranzaxis v0.0.1 (unauthenticated)

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider -/tranzaxis:
provider registry.terraform.io/-/tranzaxis was not found in any of the search
locations

- terraform.d

Expected Behavior

terraform init succeeds

Actual Behavior

Terraform tries to download the provider from registry.terraform.io

[...]
- Finding latest version of -/tranzaxis...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider -/tranzaxis:
provider registry.terraform.io/-/tranzaxis was not found in any of the search
locations

- terraform.d

It's reading some provider info from the state

$ terraform providers

Providers required by configuration:
.
└── provider[local/txr/tranzaxis] 0.0.1

Providers required by state:

    provider[registry.terraform.io/-/tranzaxis]

although I'm using a local provider

I've tried to init on a clean state and it works as expected, the applied resource has a provider reference that points to the local source that the old state doesn't

old state:

{
  [...]
 "resources": [
    {
      "mode": "managed",
      "type": "tranzaxis_instance_state",
      "name": "instance_1",
      "provider": "provider.tranzaxis",
      "instances": [...]
    }
 ]
}

new state:

{
  [...]
 "resources": [
    {
      "mode": "managed",
      "type": "tranzaxis_instance_state",
      "name": "instance_1",
      "provider": "provider[\"registry.klarna.com/txr/tranzaxis\"]",
      "instances": [...]
    }
 ]
}

Steps to Reproduce

1. terraform init

[alisdair]

@wolviecb Thanks for the report. This looks like a different issue from the original one. Would you mind opening a new issue with those reproduction steps?

If you can confirm this happens with a minimal configuration, providing that config would help a great deal as well. Thank you!

[wolviecb]

@alisdair sure, I will open another issue thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.