Terraform: Terraform Bundle is broken on master

[tomelliff]

It looks like https://github.com/hashicorp/terraform/commit/bdaf8290b458794b93cc9540de7231e655950f07 may have broken the Terraform-Bundle tool that I use to provide CI containers with all the providers we use without needing to go and grab them from the internet every time they run (plus init every location).

Expected Behavior

Given a bundle config file such as this example:

terraform {
  version = "0.11.8"
}

providers {
  aws        = ["= 1.41.0"]
  null       = ["~> 1.0"]
  random     = ["~> 1.0"]
  template   = ["~> 1.0"]
  postgresql = ["= 0.1.0"]
}

Running terraform-bundle package terraform-bundle.hcl should download the specified Terraform binary and all the listed providers at the specified version.

Actual Behavior

Fetching Terraform 0.11.8 core package...
Fetching 3rd party plugins in directory: ./plugins
- Resolving "random" provider (~> 1.0)...
- Checking for provider plugin on https://releases.hashicorp.com...
- Failed to resolve random provider ~> 1.0: host registry.terraform.io does not provide providers

Steps to Reproduce

I use the following Dockerfile to create our CI image:

FROM golang:alpine AS terraform-bundler-build

RUN apk --no-cache add git unzip && \
    go get -d -v github.com/hashicorp/terraform && \
    go install ./src/github.com/hashicorp/terraform/tools/terraform-bundle

COPY terraform-bundle.hcl .

RUN terraform-bundle package terraform-bundle.hcl && \
    mkdir -p terraform-bundle && \
    unzip -d terraform-bundle terraform_*.zip

####################

FROM python:alpine

RUN apk add --no-cache ca-certificates git make && \
    pip install awscli && \
    update-ca-certificates

COPY --from=terraform-bundler-build /go/terraform-bundle/* /usr/local/bin/

This version that checks out v0.11.8 and runs the Terraform-Bundle tool works fine still:

FROM golang:alpine AS terraform-bundler-build

RUN apk --no-cache add git unzip && \
    go get -d -v github.com/hashicorp/terraform && \
    cd src/github.com/hashicorp/terraform && \
    git checkout v0.11.8 && \
    go install ./tools/terraform-bundle

COPY terraform-bundle.hcl .

RUN terraform-bundle package terraform-bundle.hcl && \
    mkdir -p terraform-bundle && \
    unzip -d terraform-bundle terraform_*.zip

####################

FROM python:alpine

RUN apk add --no-cache ca-certificates git make && \
    pip install awscli && \
    update-ca-certificates

COPY --from=terraform-bundler-build /go/terraform-bundle/* /usr/local/bin/

References

• https://github.com/hashicorp/terraform/commit/bdaf8290b458794b93cc9540de7231e655950f07

[apparentlymart]

Hi @tomelliff! Sorry for the problems here.

The master branch currently has changes in progress for the forthcoming 0.12 release, which will include installing providers from registry.terraform.io. The server side part of this is not yet deployed since there are not yet any 0.12-compatible official provider releases (they will come at some point prior to the final release.)

Unfortunately that does indeed mean that the Terraform Core code that terraform-bundle relies on is not functional right now. I think the best approach until v0.12 is finished would be to use terraform-bundle from the v0.11.9 tag, similar to what you did here. The master branch version will become functional again once all of the providers have made at least one release published via the new protocol, so it should be possible to return to using the master branch after v0.12 final is cut from it, and probably slightly before that in practice.

Sorry for the weird behavior here. We do usually aim to keep master in a working state but the v0.12 release has enough moving parts that we needed to make some compromises in order to cut the alpha1 release ahead of the completion of some work happening elsewhere. We'll look at adding a note to the terraform-bundle documentation to suggest building it from the latest tag for now.

[tomelliff]

Fair enough, looks like I just caught things at a bad time. Thanks for the quick reply on this and looking forward to 0.12 :)

[mikecook]

I would like to point out that you've broken the tool that allows us to specify terraform and provider versions in favor of an unreleased version of terraform.

Is there no way to import the existing released versions of providers into your new registry? Since the whole point of this tool is to specify versions, it would seem that for this tool to continue to be useful you'll need to pull in older provider releases anyway...

[apparentlymart]

Because of the change in installation method for 0.12, there are now effectively two versions of terraform-bundle. The one in the v0.11 branch can build bundles for v0.11 and earlier, while the one in the master branch can build bundles for v0.12 and later.

The changes here are ultimately heading towards terraform-bundle being altogether unnecessary by instead allowing installation of third-party plugins via the registry protocol, so this non-ideal situation is a temporary compromise to get us to the ultimate goal of this tool not needing to exist at all. Terraform v0.12.0 will still only support installing the "official" set of plugins automatically, but installation of other providers from the registry will be the next step afterward. This is a necessary tradeoff to avoid delaying the v0.12.0 release even longer by adding third-party module installation to its scope.

[leeschumacher]

this is a terrible state to leave things in. terraform-build has no versioning (you haven't even updated the version string at head since this breaking change), which makes the behavior at head broken for all existing installations. Please provide better error messages and/or documentation.

[leeschumacher]

awesome, thanks for doing this!

[ozbillwang]

@leecalcote @apparentlymart

Does the latest merge fix the issue? Looks it doesn't

I can run terraform-bundle with latest terraform 0.12.2, to install the providers plugins

bash-4.4# pwd
/usr/local/bin

bash-4.4# ls -l 
total 362344
-rwxrwxr-x    1 root     root      48990304 Jun 17 04:15 terraform
-rwxr-xr-x    1 root     root      20476256 Jun 17 04:15 terraform-provider-archive_v1.2.2_x4
-rwxr-xr-x    1 root     root     145921280 Jun 17 04:15 terraform-provider-aws_v2.15.0_x4
-rwxr-xr-x    1 root     root      21828448 Jun 17 04:15 terraform-provider-dns_v2.1.1_x4
-rwxr-xr-x    1 root     root      20427104 Jun 17 04:15 terraform-provider-external_v1.1.2_x4
-rwxr-xr-x    1 root     root      20390240 Jun 17 04:15 terraform-provider-local_v1.2.2_x4
-rwxr-xr-x    1 root     root      20373856 Jun 17 04:15 terraform-provider-null_v2.1.2_x4
-rwxr-xr-x    1 root     root      20812960 Jun 17 04:15 terraform-provider-template_v2.1.2_x4
-rwxr-xr-x    1 root     root      20674272 Jun 17 04:15 terraform-provider-tls_v2.0.1_x4

But I got error

bash-4.4# terraform providers

Error: No configuration files

The directory /usr/local/bin contains no Terraform configuration files.

This is the hcl for reference

$ cat terraform-bundle.hcl
# unsupport providers for terraform v0.12.x:
#   docker=["1.2.0"]

terraform { 
  version="0.12.2" 
} 

providers { 
  template=["2.1.2"] 
  local=["1.2.2"] 
  null=["2.1.2"] 
  archive=["1.2.2"] 
  external=["1.1.2"] 
  dns=["2.1.1"] 
  tls=["2.0.1"] 
  aws=["2.15.0"] 
}

I am following this url to install the plugins in docker.

https://stackoverflow.com/a/50944611/498256

[apparentlymart]

Hi @ozbillwang,

The error message is reporting that you are running terraform providers in /usr/local/bin, where there are presumably no .tf files. Try running it in a directory containing at least one .tf file -- ideally one that refers to one of the providers you want to test.

[leecalcote]

cc @leeschumacher

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.