Terraform: Support alternate plugin release download hosts

[adamdecaf]

I noticed in reading the v0.10.0-beta changelog that plugins are now downloaded from a remote source when terraform init is ran. I have a concern in that in the context which the team I'm on is running terraform we need to keep a copy of those plugins internal to our networks. (Similar to adding an internal apt cache/proxy.)

I'd like to suggest adding some flag to use a different releaseHost in plugin/discovery/get.go.

I could think of a few options right now:

• Adding a field to the terraform provider. (similar to terraform.required_version
• Checking a -var/env variable (TF_PLUGIN_RELEASE_HOST)

[apparentlymart]

Hi @adamdecaf! Thanks for this feedback.

We're still bringing together all the documentation for this change, so apologies that things are not super clear right now.

We are not planning to support an alternate host for downloads in the first release, because we want to introduce such complexity gradually so we can make sure this feature is robust before making it super-generic, but we will continue to support the previous mechanisms for _manually_ installing providers locally, along with some new but similar conventions, and thus enable skipping the auto-download mechanism altogether when running in a production environment.

More details on this will come soon, as we finish up the documentation for the release.

[JELaVallee]

Duplicate of https://github.com/hashicorp/terraform/issues/15997 ?

[apparentlymart]

Hi all!

It's been a while, so I just wanted to share an update on the situation here.

As noted before, using the existing infrastructure on releases.hashicorp.com was a compromise we made during development in order to meet the use-case of allowing the official providers to have a separate release cycle from core without waiting until a fully-general solution was designed an implemented.

Knowing that reaching out to a remote service would not be acceptable for all users, we implemented the -plugin-dir option to terraform init as a way to disable this and instead force installation from a manually-maintained local set of providers. This then mimics the earlier usage of Terraform where separate providers were included in the main Terraform .zip file, though admittedly with reduced convenience due to the need to download each desired plugin separately.

At the time of my original comments on this subject there were no specific plans to talk about. In the mean time, we have made a plan to define a protocol that Terraform will expect for a "provider registry", similar in principle to the one already defined for module registries. With that done, we will then change the HashiCorp-distributed providers to be available via that protocol, and invite others to run their own registry servers for internal use.

As well as allowing alternative sources for the HashiCorp-distributed providers, this will also allow automatic installation of third-party providers.

The Terraform Core team is currently focused on improvements to the configuration language and so will not be able to execute on this plan immediately. However, we have begun conversations with other teams within HashiCorp to figure out all the details of implementing and deploying the new distribution infrastructure and plan to roll this out at some point after the current configuration language work is concluded.

I'm sorry that this is a bit of a "no-op" update, but I just wanted to be explicit that this is still a goal and merely a matter of resources/prioritization rather than us not intending to address this. In the mean time, I hope the -plugin-dir option continues to serve as an alternative for most cases, acknowledging that having to copy the various providers onto local disk ahead of time is less convenient than the auto-installation behavior.

[adamdecaf]

In the mean time, we have made a plan to define a protocol that Terraform will expect for a "provider registry"

Sounds good. -plugin-dir is an option that works in the meantime. Thanks for the update.

[chtompki]

Pull requests welcome?

[JELaVallee]

Thanks for the update @apparentlymart ... This is currently a big deal at my enterprise day-job where there's next to zero chance of us getting a firewall/proxy hole to releases.hashicorp.com

Ideally we'd proxy this through something like artifactory or similar caching-repo interface, so we can also leverage security and checksum scanning and the likes.

Is there anything resembling a rough ETD/release-target on this? Currently looking at a option to use -plugin-dir but we have a lot of automation using terraform init that will be adversely affected by the move from 0.9.x to 0.11.x. Will get there...

[apparentlymart]

Unfortunately this is one situation where the blocker is not implementation effort but rather design effort and coordination with other teams, since we need to define something that we're going to be able to support in the long term after introducing new features such as third-party plugin installation. Therefore I would ask community members not to propose PRs for this feature for the moment until we're able to better-define the requirements/protocol.

Our iterative development style means we're not able to predict the timing of anything other than our current tasks, but the configuration work will take at least a few more months which hopefully gives a sense of the _soonest possible_ time to start this work, which of course does not consider the amount of time it would take to actually implement it since we cannot know that until the design is fully complete.

---

For situations where an explicit -plugin-dir is hard to implement, a partial solution is to place providers in one of the directories where Terraform naturally searches for plugins, such as in the same directory as the terraform provider itself. In that case, Terraform will use a locally-available plugin if one is available that meets the version constraint given in configuration, but will still attempt to reach releases.hashicorp.com _if no matching local provider is available_.

-plugin-dir disables that latter behavior, causing an immediate error if the local directory is not sufficient, but if releases.hashicorp.com is firewalled anyway then this is just trading one error message (can't reach releases.hashicorp.com) for another (local directory does not have a version available).

This implicit approach requires careful use of the provider version constraints feature to ensure that selected versions remain consistent with what is available locally, which may or may not be more convenient than adding -plugin-dir to the init step in automation scripts.

[nhoughto]

Seems like a reasonable intermediate step to allow the override of the releaseHost value until further protocol definition etc is completed and private plugin registry is properly supported.
I'd assume this would eventually be rolled into Terraform Enterprise for on-premise private registry, which would still then require a way to specify a different releaseHost so allowing it as a first step is still a step in the right direction?

[mildwonkey]

Hello!
There have been some questions in a few different places so I wanted to post a quick update.

The terraform 0.12 release (anticipated early next year) will include support for terraform providers in the registry. In the first release this will support our official providers, (those hosted on releases.hashicorp.com), and over time we will add support for community providers and alternate release hosts or sources (the details have yet to be determined).

Cheers!

[sontags]

Hi all,

Has there ever been a discussion about implementing a source into the provider itself? I would love to see something like...

provider "[custom-provider]" {
  source {
    arch =  "linux_amd64"
    location = "https://[some-url]" 
    md5sum = "[some-sum]"
  }
  source {
    arch =  "win_amd64"
    location = "https://[some-url]" 
    md5sum = "[some-sum]"
  }
  ...
}

terraform init would fetch the source, check the sum and place the binary in .terraform/plugins.

This pattern would allow to have custom providers (we use some very specific providers internally) installed by same mechanism which are used when using 'standard' providers without having additional infra (such as a registry, since s3 for example would do).

_EDIT: I got a bit off topic here, I'll open up a separate issue..._

[bodgit]

Just being able to override the base URL would be sufficient here. I don't particularly want/need to spin up a private registry as I already have releases.hashicorp.com set up as a generic repository in Artifactory so it's effectively a pull-through proxy/cache. Being able to override http(s)://releases.hashicorp.com/ with https://artifactory.example.com/path/to/repository/ would be enough to get things working.

[Ray-B]

Would it be possible to get an update or additional clarification on the state of this work? Last update was 5 months ago. We're now into the better part of Q2.

More specifically, would be interested in suggestions on how to actually make use of _-plugin-dir_ from end-to-end.

To simply say "put the custom provider in this dir" is a bit...vague. Are there any recommendations on how to make this approach salable? Are people just pre-install their private provider on docker containers? or some other strategy? What about using S3 buckets? etc.

[cullenmcdermott]

If someone were to submit a PR that allowed optionally overriding the releaseHost var with an environment variable, would that PR have a chance of being accepted? Our use case is simple, we would just like to point this at our internal mirror of releases.hashicorp.com.

[apparentlymart]

Terraform v0.12 (and thus current master) no longer uses releases.hashicorp.com to locate providers. It uses registry.terraform.io instead, which is the first step towards supporting installation from other "registries" (really: any HTTP server that can return suitable metadata to find the necessary binaries). Configuration mechanisms to explicitly specify other installation sources will follow, once we've done some other necessary work to make sure that it can support other signing keys, and other such things that are required for having an open ecosystem.

However, if your requirement is just to override the location of the primary "registry" to point at a local proxy, Terraform v0.12 will already support this via a .terraformrc setting that isn't yet documented, but should be documented soon:

host "registry.terraform.io" {
  services = {
    "modules.v1": "https://your-mirror.example.com/v1/modules/",
    "providers.v1": "https://your-mirror.example.com/v1/providers/",
  }
}

The services argument here overrides what would normally fetched by the service discovery protocol, so in this particular case you'd be telling Terraform to use the above result instead of what it would normally have fetched from https://registry.terraform.io/.well-known/terraform.json. If you only want to reroute the provider installation protocol and continue to get modules from the "real" registry.terraform.io then you can set modules.v1 to https://registry.terraform.io/v1/modules/ to mimic what that host would itself have returned.

Note that the registry is just an _index_, not the hostname where the providers archives are distributed from. For the moment, that'll still be releases.hashicorp.com, but is likely to transition somewhere else over time, and for third-party providers (once supported) could be somewhere else entirely, like GitHub releases. To actually serve different packages, then, your mirror would need to return different URLs from the download location endpoints. The providers.v1 protocol isn't publicly documented yet, but documentation for it will be published as part of finishing up this mechanism so that third-party registries can also be supported.

This issue is, for us, representing the capability to install from third-party registries at _different_ hostnames, rather than overriding the meaning of registry.terraform.io itself (so users can mix both "official" and third-party providers easily in the same configuration), and that will take some more work, but the above host services override mechanism will be present in v0.12.0 at release because this mechanism was originally introduced to allow a similar redirection capability for _module_ installation, in Terraform 0.11.

[adamdecaf]

Thanks for the updates @apparentlymart!

[ringods]

@apparentlymart with v0.12 pointing to registry.terraform.io, can one also deploy third-party providers there?

As part of the terra-farm organisation, we recently deployed our first module on the registry. On Github, we also have a few third-party providers we would like to publish in an easy way.

[apparentlymart]

The server-side implementation and signature-checking for third-party providers on registry.terraform.io is not yet in place, so at the time I write this the public Terraform Registry only indexes the same plugins as on releases.hashicorp.com, but indeed the intent is to support third-party providers via the public registry in a later release, as well as installation from other hosts entirely if they implement the provider registry protocol. That will allow both automatic installation of community-maintained open source providers (which can be posted on the public registry) and use of proprietary internal providers hosted on your own private registry, with the latter being possible without mirroring the whole public registry onto a local host.

The work for this generalization is primarily being driven by the team that maintains Terraform Registry (rather than the Terraform Core team) so I don't know when it will be completed in terms of Terraform Core release versions, but I can say that the work is planned out in terms of high-level steps and this initial release supporting only the HashiCorp-distributed providers is one of the first steps in the plan.

[oonisim]

As per @ cullenmcdermott:

Our use case is simple, we would just like to point this at our internal mirror of releases.hashicorp.com.

It is such a pity that simply because we cannot use the internal (no direct internet) mirror, we need take Terraform off from the tool selection.

[mafrosis]

@oonisim @apparentlymart says in his earlier https://github.com/hashicorp/terraform/issues/15252#issuecomment-488781567 that this is possible with Terraform v0.12.

My bad - that just changes the Index URL. There is no way to override the actual base URL where binaries are pulled from.

[Techcadia]

So I have been looking at hosting a custom registry for a few internal providers.

After digging pretty deep into how its fully implemented and what is required. I stumbled onto this issue https://github.com/hashicorp/terraform/issues/20527 Which requested that the gpg key that originally was pinned inside the bin was reinstated. That was completed in this PR https://github.com/hashicorp/terraform/pull/20543

As it currently stands no one except HashiCorp public key can publish towards any registry including custom ones because of the gpg signing. Is there any goals to allow even a .terraformrc config to turn off this gpg check while keeping the registry gpg check in place.

[adamdecaf]

Is there any goals to allow even a .terraformrc config to turn off this gpg check while keeping the registry gpg check in place.

Why not a config to specify a different public key?

[Techcadia]

@adamdecaf Just allowing a different key inherently means that you have to retrieve and download every provider and then resign with that key. I personally would like to only have to host and sign the providers I maintain.

The real goal is establishing trust with the registry (public/private) and confirming that its not been compromised.

[circa10a]

@mildwonkey any updates?

[chtompki]

@circa10a - fair point. This thread is going on 3 years old with a fairly reasonable request

[mildwonkey]

👋 Hi everyone! I finally have an update on this long-awaited feature for y'all.

Back in January we announced our plans for the provider "source" attribute: https://www.hashicorp.com/blog/adding-provider-source-to-hashicorp-terraform

And last week, we announced our v0.13 beta release: https://discuss.hashicorp.com/t/terraform-v0-13-0-beta-program/9066
https://github.com/hashicorp/terraform/issues/25016

This new source attribute will allow you to specify alternate registry hosts for any given provider.

I also see some questions regarding the GPG signatures. The registry response includes a GPG public key and Terraform will use that key to verify providers which aren't signed by HashiCorp. Terraform will display a warning notifying you that the provider binary is self-signed.

We are expecting to ship beta 1 on June 3rd, and I'd like to encourage y'all to give it a try (on non-production infrastructure, please!) and let us know what you think and if you run across any issues.

Thanks for you patience - it's been a long project, and there's work still to be done for future releases, but I'm excited to finally get to share what I and my team have been working on for the past year.

[smiller171]

@mildwonkey When can we expect more documentation on this?

Also I noticed the new format has been backported to 0.12. Will 0.12 actually be able to utilize other sources or is this just for code compatibility?

[mildwonkey]

Both good questions! This feature will not be available in terraform v0.12: we slowly started introducing the new code in v0.12, but terraform did not (and will not) start parsing source attributes until v0.13.

The documentation will get deployed to our website when we release Terraform v0.13 (General Release), tentatively scheduled for 6 weeks after the first beta. More information and documentation will be shared as we get closer to the beta release.

[TechIsCool]

So I dug into the code and really enjoy what I see https://github.com/hashicorp/terraform/commit/22ef5cc99cb8bf06040dd8e56b715a835e31b913

I am curious if we will be able to pin a (Internal/Company) GPG key and disable self-signed packages.

The only reason this functionality didn't make it into 0.12.x was because the registry was the source of truth for both package and registry. The Concern in #20527 is still valid for a MITM attacks. For example I modify a provider to target a specific resource. The only difference to an end user is a warning about a provider not being signed.

[smiller171]

So I dug into the code and really enjoy what I see 22ef5cc

I am curious if we will be able to pin a (Internal/Company) GPG key and disable self-signed packages.

The only reason this functionality didn't make it into 0.12.x was because the registry was the source of truth for both package and registry. The Concern in #20527 is still valid for a MITM attacks. For example I modify a provider to target a specific resource. The only difference to an end user is a warning about a provider not being signed.

If it's not officially supported you could only run TF from CI and have the CI fail if it sees that warning in the plan. Obviously native support would be better though.

[apparentlymart]

Hi all,

There's a lot of discussion here from along the way while this was under design and development, but now that the first iteration of this feature is merged I'm going to close this out so we can fan-out further discussion and feedback into more focused separate issues. If you have feedback after you try out this feature either by compiling from source right now or by using the forthcoming beta releases, please feel free to open an new specific issue and then we'll be better able to engage with each conversation separately.

Thanks!

[danieldreier]

Terraform 0.13.0 beta 1 launched today, and I'd love to get feedback from people who have been following this issue. I hope you'll try it!