Terraform: What can folks assume when they derive new Docker images from the official Terraform Docker images?

[xczimi]

Terraform Version

0.12.27
0.12.28

Terraform Configuration Files

n/a

Debug Output

Crash Output

Expected Behavior

# docker run -ti --entrypoint /bin/cat hashicorp/terraform:0.12.28 /etc/alpine-release
3.11.6

# docker run -ti --entrypoint /bin/cat hashicorp/terraform:0.12.27 /etc/alpine-release
3.11.6

# docker run -ti --entrypoint /bin/cat hashicorp/terraform:0.12.26 /etc/alpine-release
3.11.6

Actual Behavior

# docker run -ti --entrypoint /bin/cat hashicorp/terraform:0.12.28 /etc/alpine-release
3.9.2

# docker run -ti --entrypoint /bin/cat hashicorp/terraform:0.12.27 /etc/alpine-release
3.9.2

# docker run -ti --entrypoint /bin/cat hashicorp/terraform:0.12.26 /etc/alpine-release
3.11.6

Steps to Reproduce

See behaviour

Additional Context

This makes it impossible to build CI/CD images consistently using terraform as the base image

References

[apparentlymart]

Hi @xczimi,

We consider the OS in these Docker images to be an implementation detail rather than something users ought to be relying on. Although we don't have any immediate plans to do so, as we currently stand our docker images could switch to an entirely different base distribution entirely in future if that would somehow make docker run hashicorp/terraform (that is, directly using the images we supply, rather than customizing them) work better.

Can you say a little more about what you are trying to do here and why the exact OS the image is built on is significant for that goal? I ask because I'd like to rework this issue into an enhancement request to meet your use-case, which I expect will involve putting some constraints both on us (constraining how the docker image build process can evolve in future) and on you (constraining how you should use that image if you want what you are doing to be reliable for future releases), but I want to understand exactly what you are aiming to do here so we can make a suitable tradeoff to that end.

Thanks!

[xczimi]

Hi @apparentlymart,

I understand you only provide these images as a means to execute the terraform binary via docker run hashicorp/terraform, from this point of view, the base of your image is indeed an implementation detail.

However, docker images meant to be use as layers or building blocks and since our primary tool in our infrastructure deployment was terraform using hashicorp/terraform as the first layer of our build container made sense.
This allows us to use the latest stable version pretty much as soon as they are released.

#infrabuild.Dockerfile
FROM hashicorp/terraform:0.12.26

RUN apk add --update git openssh python3 py3-pip jq util-linux nodejs npm
RUN pip3 install --upgrade awscli
RUN npm install -g yarn

From this point of view, the base layers of your docker image is never really just implementation detail as it's part of the image you publish.

Looking around the codebase, based on the history of scripts/docker-release/Dockerfile-release

This may ultimately be a corrected historical problem in the docker image publish process as the file explicitly uses alpine:3.9.2

0.12.1  3.9.2
0.12.2  3.9.2
0.12.3  3.9.2
0.12.4  3.9.2
0.12.5  3.10.1
0.12.6  3.10.1
0.12.7  3.10.2
0.12.8  3.10.2
0.12.9  3.10.2
0.12.10  3.10.2
0.12.11  3.10.2
0.12.12  3.10.2
0.12.13  3.10.3
0.12.14  3.10.3
0.12.15  3.10.3
0.12.16  3.10.3
0.12.17  3.10.3
0.12.18  3.10.3
0.12.19  3.11.2
0.12.20  3.11.3
0.12.21  3.11.3
[...]
0.12.23  3.11.3
0.12.24  3.11.3
0.12.25  3.11.6
0.12.26  3.11.6
0.12.27  3.9.2
0.12.28  3.9.2

I still think changing the base image is significant change to warrant a note in the changelog for a release.

[tklovett]

Ditto on the use case of using dockerized Terraform as a building block to create our custom toolchain. We start hashicorp/terraform:$version, add Terragrunt, Scenery, terraform-provider-sops, etc. For these steps, we must have knowledge that the underlying platform is linux_amx64.

From there, we apk add jq=$version python3=$version (etc), with versions pinned for stability. We enforce version pinning via hadolint rule DL3018. On bumping the version of the underlying hashicorp/terraform image, I discovered that I would have to downgrade the pinned APK versions.

I understand that the intent was for the platform to be a hidden implementation detail, but in practice this ended up being a backwards-incompatible change.

[xczimi]

From there, we apk add jq=$version python3=$version (etc), with versions pinned for stability. We enforce version pinning via hadolint rule DL3018. On bumping the version of the underlying hashicorp/terraform image, I discovered that I would have to downgrade the pinned APK versions.

In our case, it wasn't even a version downgrade, it resulted in a missing package introduced in alpine v3.11: docker-cli

[apparentlymart]

Hi all,

Thanks for sharing this additional information.

I think the path forward here is to refine the definition of what public interface this image is providing. As we currently stand, the only public interface it is providing is that you can run terraform in it and that it has git installed for module installation.

The opposite extreme would be to say that the image is totally frozen apart from the version of Terraform installed in it, but that is an untenable extreme because we will need to make changes to the base image over time as Terraform evolves and as Alpine Linux evolves.

So we need to find a compromise here, where certain other aspects of this image that folks are relying on will be considered part of the contract and thus be considered breaking changes if they are altered in a future release. The Docker ecosystem doesn't provide any formal way to express such an interface, so I suppose we'll need to write it down as prose in the image's README once we've defined it.

We won't be able to work on this in the near future because our focus is on features inside the product as consumed by the primary release channel (the packages on the Downloads page), and not on the secondary distribution channels like Docker.

Until we're able to define such a thing, you should assume that anything about this images can change at any time except that they run Terraform. If you are building a custom image then for the moment I'd suggest copying Terraform's release dockerfile and altering it to suit your needs, rather than using its resulting image as a FROM. Terraform is a statically-linked executable, so there's nothing particularly special required to install it into a Docker image except to make sure that you have available any version control systems you intend to use to install modules.

[xczimi]

For starters I think the image published to docker hub should match what the source code defines, which was and still is alpine:3.9.2 according to scripts/docker-release/Dockerfile-release. Looks like it'll get an upgrade for 0.13 which is great

e.g. https://github.com/hashicorp/terraform/blob/v0.12.26/scripts/docker-release/Dockerfile-release defines the base image to be alpine 3.9 , but that's not the image published to docker hub is.

If that stays consistent this wouldn't really be an issue

[alvin-huang]

Hi @xczimi. The cause of the issue here is there was a script we used a separate Dockerfile in the build pipeline to build releases rather than the one in the Terraform repository. When we migrated CI providers, we chose to use the Dockerfile in the hashicorp/terraform directory under scripts/docker-release/Dockerfile-release as you have found. This caused the mismatch of the one in our other Dockerfile using alpine:latest while the one in the repo had a pinned version.

The fix in #25456 will bump all Docker images moving forward to use alpine:latest as a base image so there will not be unexpected downgrades. However, since alpine:latest is not a pinned version, you can expect _upgrades_ which may cause issues of packages being available/unavailable in newer base images. If this causes issues in the future, I would suggest pinning package installations. We may also switch to other base distributions in the future as both Terraform and Alpine evolve, and so for now we're making no additional promises about how these these images might change in future.

The official artifact we provide to users is our binary on releases.hashicorp.com and the Docker image is for convenience. To mitigate any risk you may have in your custom image I would suggest using our image in a multi-stage docker build to copy the binary out as @apparentlymart has suggested or directly installing the binary from releases.hashicorp.com, rather than using our image as the base.

The 0.12.27 and 0.12.28 images have been updated:

and they use alpine 3.12.0:

docker run --entrypoint="" hashicorp/terraform:0.12.28 cat /etc/alpine-release
3.12.0

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.