Terraform: Provide a CLI way to download the latest version of a provider globally

[ColinHebert]

Currently if I want to install a provider globally, I need to:

• Go on https://releases.hashicorp.com/
• Find the plugin I wish to install
• Find which version is the current latest (there is no lookup key, so I need to manually compute that information)
• Find the version that matches my operating system
• Download that version on disk
• Extract it on disk in a specific location (say /usr/lib/terraform/my_provider-1.0)
• Create a "providers" entry in ~/.terraformrc if it doesn't exist
• Add an entry "my_provider" with the path of the downloaded executable if the entry doesn't already exist. Otherwise I need to update the entry.

All those steps get in the way of an easy automation. The good news is, most of that work has already been done with the current implementation of the terraform init command.

Would it make sense to have a slightly improved plugin/provider management that allows the installation of a provider globally; something along the lines of terraform plugin install my_provider (with the relevant options to update to latest, pin to a version, force an operating system, etc.)?

[jbardin]

Hi @ColinHebert,

We definitely want to make running Terraform in automation a first-class use case.

Installing "globally" however is often at odds with automation, as there is often no "global" context shared between executions. Also, the use of the legacy provider entries in .terraformrc aren't recommended, because they cannot be properly versioned.

We currently provide 2 primary methods of "vendoring" plugins:

• manually download the required binaries into terraform.d/plugins/OS_ARCH in the configuration directory. This directory is automatically checked during the plugin discovery process, and is primarily used for locating plugins outside of the default set available with Terraform.
• Store the plugins in a location of your choosing, and provide the -plugin-dir flag to the init command. This will override the entire automatic installation process, only using the plugins provided.

Having a command to populate a directory with a defined set of plugins may still be useful, so we can keep this open as a feature request while we work on improving the workflow.

[ColinHebert]

Installing "globally" however is often at odds with automation, as there is often no "global" context shared between executions

I beg to differ, multiple dependency management systems already solved that global context problem, whether it's pip, gem, maven, etc.. Having the providers in a single location doesn't imply that each of them needs to be loaded nor that you can't have multiple versions of a same provider sitting next to each other. While the decision of which one should be loaded is delegated to the configuration of the project itself.

We currently provide 2 primary methods of "vendoring" plugins:

From my understanding both those options are entirely manual and require human intervention to handle not only the version being used but also other details such as the platform etc.

the use of the legacy provider entries in .terraformrc aren't recommended, because they cannot be properly versioned

This is what the current documentation seems to recommend when it comes to solving that specific issue. https://www.terraform.io/docs/plugins/basics.html#installing-a-plugin

Having a command to populate a directory with a defined set of plugins may still be useful, so we can keep this open as a feature request while we work on improving the workflow.

I'd agree with that, in particular having a way to mimic the behaviour that is already available without the -plugin-dir option with a custom location.

[jbardin]

I beg to differ, multiple dependency management systems already solved that global context problem

Fair enough. I probably shouldn't have stated it as global being bad, I was thinking of the implied global aspect of ~/.terraform* files which aren't correlated with individual configurations. An automation "user" may have multiple configurations to operate on, which require multiple provider versions.

This is why we are providing individual vendoring locations within the config, or the ability to specify them explicitly with -plugin-dir.

Thanks for pointing out the plugin/basics page. It's outdated, and we will get that fixed.

We are also maintaining a new page specifically for this use-case:

https://www.terraform.io/guides/running-terraform-in-automation.html

[ColinHebert]

I've read the Running Terraform in Automation guide, unfortunately (and ironically), the solution to the "Pre-installed Plugins" problem is to do the manual operations I was mentioning earlier which means that in order to get my automation to work I need to manually bootstrap the system (ie install the plugins) through a fairly tedious process which would take a fair bit of effort to automate.

There are in the mean time things that could be done to improve that process.
An example would be eliminating the "Find which version is the current latest" step by providing a "latest" link that would allow users to download whichever version is the latest (could be done through HTTP redirection to keep the version in the filename).
Or removing the -plugin-dir step by having a default (convention over configuration) directory in which can be used as a fallback if the provider isn't available in the local .terraform folder.
Another option would be to allow terraform init with the -plugin-dir to download plugins in the given plugin dir (removing the entire download/extract workflow).

[nbering]

So far I haven't seen anyone say it's a bad idea to provide a more convenient way to install plugins outside the individual config context, it just doesn't exist yet. It seems to me that an immense amount of effort went into just getting the providers separated from core. Maybe just be patient and this will come along? I would also like to see some plugin management commands at some point, so maybe we can get a community RFC together...

[apparentlymart]

Hi all! Thanks for this great discussion.

The auto-install behavior as currently implemented is focused on the per-project directory for a few reasons. The initial reason is that when people are getting started with Terraform we assume they would rather it not make global changes. However, a secondary reason is that we assumed that people installing plugins in a _global_ location would be doing it with an existing config management tool, most of which have existing support for fetching archives from servers and installing them.

As an aid to those who want _not_ to depend on the official releases server from their config management (that is, want to download once and then install from a local server) we built a little tool to automatically produce "bundle" archives. At present this tool is separate from Terraform, but that's not for any strong reason. We could potentially implement functionality like this in the main Terraform binary, since indeed it is running the same code as terraform init uses for installation.

I'm curious to hear what sort of thing people are trying to do here. Is the idea just to populate a directory locally to easily share between multiple working directories, or is it installing on servers with config management? (or, indeed, something else!)

[endofcake]

An example use case: we deploy Terraform from our CI using a Docker image (based off the official one). The only provider that we really care about is aws, so it would make sense to pre-install this in the image globally rather than do terraform init on every build. At the moment it seems like it's possible, but we'd have to jump through a fair number of hoops to do this.

[woohgit]

@ColinHebert What I don't get is: Why the -plugin-dir option xor with the -get-plugins option? I'd love to able to define a custom path for my automatically installed/upgradable plugins and share it inside my environment. We want to use 1 plugin dir / repository which is now impossible. A good solution would be to allow the -get-plugins=true if the -plugin-dir is set.

So what I would expect:

running terraform init -plugin-dir ../common/plugins -get-plugins=true would automatically download/upgrade the required plugins to ../common/plugins and not to the .terraform/plugins.

I think this use-case would be very useful for everyone.

What do you think? /cc @jbardin

[pdecat]

As a terraform configuration maintainer, I'm using strict terraform and provider versions constraints to pin them and get reproducible results (and avoid surprises when behavior changes from one minor release to another), for example:

terraform {
  required_version = "0.10.2"
}

provider "aws" {
  version = "0.1.4"
}

In that situation, a related use case where automation would help is to able to:

• list available provider updates (kind of like terraform version notifying of terraform versions updates),
• selectively update provider version constraints in local .tf files,
• test the provider upgrade locally before committing the version constraint change

[lestephane]

@apparentlymart Thanks for the tip about terraform-bundle, I've used it in a docker image to not pollute my host OS. It produces a zip which i can just unpack in /usr/local/bin in another command-and-control vm where i run my terraform commands. It works great so far, and i only have a single copy of the plugins lying around, regardless of how many config directories i have. The gist is at: https://gist.github.com/lestephane/495f35de9a398ad0f613458aaeb9db15 (in case anyone is interested)

[DanielJonesEB]

@woohgit's suggestion would be helpful for our use case, where we want to be able to provide custom plugins _and_ download any of the standard ones that might be required.

[shantanugadgil]

I am all for some simple method to download the needed (all?) plugins.
_(stumbled upon this issue)_

For now, what I do is:
on a central server, I run "terraform init" periodically and copy out the files from .terraform to bundle along with the terraform binary, where I actually need them.

[apparentlymart]

A while back we added the optional provider plugin cache that causes Terraform to populate downloaded files in a separate (possibly global) directory and then re-use those files for future runs of terraform init.

It's likely that in a future release we will take another pass at version management, to deal with issues such as integrity checks and managing intentional upgrades. Those use-cases are likely to end up belonging to #17110 along with the same for modules, since the same mechanism will probably be used for both.

[Immortalin]

Also in elastic scaling scenarios running terraform init multiple times would constantly hit hashicorp's servers and we don't want our production systems to get banned.

[pauldraper]

Web scraping to achieve https://github.com/hashicorp/terraform/issues/15801#issue-249988143 :

/usr/local/bin/latest-terraform-provider

#!/bin/sh -e
curl -LSs "https://releases.hashicorp.com/terraform-provider-$1/" 2> /dev/null \
  | grep -m1 -o "terraform-provider-${1}_[0-9.]*" \
  | cut -d_ -f2

Example:

$ latest-terraform-provider aws
2.15.0