Terraform: Separate provider version constraints from provider configuration

[apparentlymart]

When we introduced provider version constraints in 0.10 it felt natural to do this via a new version argument inside the existing provider block type, like this:

provider "example" {
  version = "~> 1.0"
}

However, in 0.11 we changed the model for how providers and modules interact to address some confusing interactions such as inheritance of aliased configurations, partial overriding, and instantiating the same module source multiple times with different provider configurations. This new model has caused us to recommend that provider blocks should appear only in a root module, and re-usable modules (which are analogous to libraries in a traditional programming language) should instead receive provider configurations as an argument, either implicitly (through inheritance) or explicitly (via the new providers argument for modules).

Due to the prior overloading of the provider block to represent both a single provider configuration and a constraint on the (global) provider version, this has effectively left reusable modules no convenient way to declare provider version constraints.

---

To rectify this, we're considering introducing a new provider version constraints mechanism within the terraform block, alongside the existing required_version attribute that constraints the version of Terraform Core:

# NOT YET IMPLEMENTED; may change before release

terraform {
  required_version = "~> 0.11"

  required_providers {
    aws    = "~> 1.1"
    consul = "~> 1.0"
  }
}

By separating the idea of a provider version constraint from the idea of a provider configuration we can resolve the architectural wart that 0.11's new approach introduced.

---

Given that the version argument for the provider block has been around for some time now we would retain it as an alternative way to make constraints, but the new approach described above would become the recommended method for specifying constraints in child modules.

Some further recommendations would be required due to the fact that each provider can only have a single version at a time for a given configuration:

• Re-usable modules should use a >= constraint for the minimum version that contains the features it requires, unless there is a known incompatibility with a later version.
• Root modules should continue to use ~> constraints to lock to a particular major or minor version (depending on the user's risk tolerance) that is at least as new as the newest minimum from a child module.

[texascloud]

That terraform block will be placed in the module right? Is it currently possible to have two providers of the same type (one with an alias) but different versions? If so, would this change work with that?

[apparentlymart]

It is not currently possible to have two providers with different versions in the same configuration, and this new approach does not change that. Instead, Terraform would continue its current behavior: find the newest available plugin that meets _all_ of the given constraints.

If two modules (or today, two provider blocks in the same module) disagree on version constraints to the extent that they are in conflict, terraform init will fail due to not being able to meet the given constraints.

[texascloud]

If a provider's version does not meet any one module's constraints for that provider as defined in its terraform block, when would the error signaling the mismatch pop up? terraform init?

[apparentlymart]

It does indeed get detected by terraform init, as part of its provider initialization step:

$ terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...

No provider "aws" plugins meet the constraint "< 1.0.0,>= 2.0.0".

The version constraint is derived from the "version" argument within the
provider "aws" block in configuration. Child modules may also apply
provider version constraints. To view the provider versions requested by each
module in the current configuration, run "terraform providers".

To proceed, the version constraints for this provider must be relaxed by
either adjusting or removing the "version" argument in the provider blocks
throughout the configuration.

$ terraform providers
.
├── provider.aws >= 2.0.0
└── provider.aws.nope < 1.0.0

[texascloud]

Does the terraform providers command list out the module names too and just isn't shown off in the above example output? That would be incredibly useful to know.

Do you think it would make sense to have some functionality as part of the providers command that could adjust the provider versions for you? Or perhaps just output a provider version such that it fits within the constraints of all modules which define a version provider. If no such version exists, that could be said as well, which would signal that your modules are wholly incompatible.

[apparentlymart]

Yes, there were not any child modules in my example configuration there but the purpose of that providers command is to show you all of the distinct version constraints across all provider blocks in all modules so that you can figure out where a conflicting requirement is coming from.

terraform init already selects one version (the newest version) that matches the given constraints, if there is at least one such version. Currently we don't have a good way to generate/amend config on the fly (the configuration language printer is not robust enough for us to be confident in using it for modifications other than pretty-printing) but this sort of thing may be possible later. For now, this issue is just about changing the way version constraints are expressed in configuration, rather than changing details about how those constraints are interpreted or managed.

[dghubble]

Would this alleviate the need to explicitly pass modules to satisfy version constraints? Taking the example in https://github.com/hashicorp/terraform/issues/16824#issuecomment-348741665, would the module instance become (removals commented):

module "aws-cluster" {  
  providers = {
    aws = "aws.myalias"
    #local = "local.default"
    #null = "null.default"
    #template = "template.default"
    #tls = "tls.default"
  }
  source = "..."
  ...

Basically, when a module in the new system declares required_providers in a terraform block, does that compare only against explicitly passed modules or against some default (i.e. alias isn't in use)?

[jbardin]

Hi @dghubble,

That's exactly the idea. The proposal allows modules to separate the provider version constraints from the provider configuration.

There can only be one version of a provider at a time, so provider version constraints are applied globally. The proposed required_providers block would allow modules to add their constraints without requiring the provider configuration blocks.

[dghubble]

Proposal sounds good to me then :)

[barotn]

Hi Guys,

When can we expect to see that feature to "Allow Variable to control provider version" ??

Thanks

[neerfri]

Is this still being considered?

[apparentlymart]

It is likely (though not confirmed) that this will be in the initial 0.12 release. Since it's a stretch goal, it may end up being deferred to a later 0.12 point release, but we definitely to still plan to do it.

[neerfri]

Thanks @apparentlymart
Is there a place to see the plans for 0.12?

[pecigonzalo]

@apparentlymart thanks for this, im hitting an issue on a submodule which had provider version set and now im unable to delete it.
What is the procedure in this cases? A targeted destroy before we remove the code?

What about modules that have multiple providers with aliases? Is the recommendation now to not have multiple providers on a single module?

[glucas]

@neerfri There is a whole series of blog posts on upcoming 0.12 changes, in case you haven't seen them: https://www.hashicorp.com/blog/category/terraform

[apparentlymart]

Hi all!

The feature I described in this issue has now been merged into master ready to be included in the forthcoming v0.12.0 release. I just verified it in the v0.12.0-alpha1 build by using the configuration example in my original comment.

$ terraform providers
.
├── provider.aws ~> 1.1
└── provider.consul ~> 1.0

Terraform v0.12.0 will now treat this new form as an alternative way to set provider version constraints. The old form with the constraint inside the provider block is still supported, and if both are set then Terraform will require both of the constraints to match.

Since this is merged and ready for release, I'm going to close out this issue. Thanks for the great discussion here!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.