Moby: Allow using ENV or ARG in `FROM`

[griwes]

I feel like the list of Dockerfile instructions that support environment variables should be extended to include FROM.

Here's the situation I'm facing right now. I'm trying to build myself a set of Docker images for C++ development - a container per used compiler version, a container per used Boost version etc. For example, I'll have gcc:4_8_5, gcc:5_1_0, gcc:5_2_0, llvm:36, llvm:37. Then, for each used version of Boost - say 1.58.0, 1.59.0 - I want to build it with each of those compilers.

So, the end result would be something like this:

• llvm-37-boost:1.58.0
• llvm-37-boost:1.59.0
• gcc-4_8_5-boost:1.58.0

etc.

Currently I can't really do this if I want to (1) fully reuse the original compiler containers and (2) not do manual templating of the boost dockerfiles myself (having a dockerfile per version of the base container doesn't seem sensible). If I could just write:

export base_image=$1 # the first argument to the build script
docker build -t $base_image_prefix-boost:$version -f boost.gcc.docker

it'd simplify this significantly, but that'd require the ability to write:

FROM ${base_image}

Has this ever been discussed? Is there a reason why environment variables aren't supported in FROM?Could such support be added?

[duglin]

I tend to think we should allow env vars in all (or most) Dockerfile cmds.
ping @tiborvass

[cpuguy83]

Where would this env var come from?
I think being able to dynamically set FROM is NOT a good idea.

If you want to generate a Dockerfile from a templating language then by all means, but supporting this in the Dockerfile is just asking for bloody feet.

[duglin]

I was thinking of something like this would be good:

ARG imageVer latest
FROM ubuntu:$imageVer
....

$docker build --build-arg imageVer=14.04 .

Then someone can use my dockerfile w/o having to modify it just to get a different version of ubuntu.

[griwes]

In the exact case I was talking about docker build is invoked from a bash script that sets the environment properly, but I think the ARG thing @duglin mentioned is actually even better than just using environment variables, as it's quite flexible. I'd like that.

[duglin]

@griwes yea, there is no -e option on build so it would have to use --build-arg. This would also require changing the build logic to allow ARG before FROM - right now there's code to make sure FROM is always first. A minor change tho.

[crosbymichael]

I'm -1 on this because dockerfiles should try their best to produce a consistent result and by making FROM variable this totally sets us on the wrong path.

[phemmer]

I'm -1 on this because dockerfiles should try their best to produce a consistent result and by making FROM variable this totally sets us on the wrong path.

Haven't we already tossed this ideology out the window with --build-arg

[MHBauer]

This is where I typically see people use a template to be able to FROM some slightly different, but otherwise varied base images. I'm not sure if that's an argument for or against this proposal.

[justsml]

ditto @crosbymichael @MHBauer
:-1:

Docker's Jedi power comes from it's immutable soul.

[duglin]

@griwes in the meantime you may try using multiple Dockefiles along with the -f flag - e.g.
FROM=abc docker build -f Dockerfile.$FROM ...
Of course, it wouldn't be nearly as bad if we supported an INCLUDE statement (#12749), then you wouldn't need to duplicate everything in the Dockerfile, but alas there's where we are at the moment.

[justsml]

You should check out docker-compose - it's designed as 'a management layer' which sits above the more focused & scoped docker-engine.

Check the link: variables in docker-compose

Here's the current example:

db:
  image: "postgres:${POSTGRES_VERSION}"

[pekra]

+1 for that feature like duglin has mentioned it in his second post, because we want to build our images for example with different tomcat and jdk versions
Would make life much easier than have a work around with shell scrips and template files...

[nakedible-p]

I will add one use case here:

We have our own custom base images on top of which each piece of software written by us runs. This is important to us as we need to have PCI compliant environments which requires us to have hardened base images. It also means we have to update every docker image we have atleast once per 30 days to get security patches in. However, we'd like to do this in a controlled manner, instead of just using "latest" as the version and not having our builds reproducible.

One way to do this would be to hardcode the base image version in the FROM clause in the Dockerfile - however, this would mean that we would need to have a separate pull in each repository (10+) each month for updating the base image version. Another way would be to have Dockerfile be a template from which we generate a suitable FROM clause to the actual Dockerfile, but we prefer concrete files instead of templates in our version control. The way we actually use is that our build scripts dynamically tag the selected base version as "latest" at build time and the FROM clause only specifies latest as the version. However, this is confusing for most developers.

A much better solution would be to be able to say ARG BASE_VERSION and FROM our-linux-base:$BASE_VERSION and then just use docker build --build-arg BASE_VERSION=1.4.0. Not specifying a base version would be an error (missing argument).

And finally, a small rant:

Docker's Jedi powers may come from its immutable soul, but specifying FROM project:latest is far from immutable! If you don't intend to disallow all such forms except FROM project@sha256:cafebabe, I hope you will consider the real world use cases that would need the feature, instead of all the bad things those on the dark side might do with it.

[jamshid]

I was also expecting ARGs to be usable in FROM. I know you don't want to enhance Dockerfile syntax but this would be very useful and consistent. It seems safe, since the FROM value is included in the image hashes, and it's a lot cleaner than adding a tempting language or using sed scripts and temp files (my solution before ARG).

ARG VERS=centos6
FROM centos:${VERS}

[aacebedo]

+1 here !

[jakirkham]

ARG was also proposed here ( https://github.com/docker/docker/issues/18633 ).

[duglin]

@justsml's reference to compose ( https://github.com/docker/docker/issues/18119#issuecomment-158818015 ) is an interesting one. The fact that Compose sees a need to introduce some variability in the images/containers it manages isn't that different from the variability image builders need. In both cases there's a realization that the inputs (Compose yml & Dockerfile) are really just a template and based on the exact environment/need we may need to tweak things slightly. Asking people to duplicate everything for each variant/case is a maintenance hassle (or nightmare). Heck, why do we even allow env vars to be passed into containers at runtime at all if want consistency/immutability - its because reality comes into play at times. Compose saw that, I think the builder needs to as well.

[bernardgut]

I have images that need different dependencies depending on which kernels they are running on. I can't understand why

FROM someapp-$(uname -r):current_release

is so far fetched for a dockerfile. Right now managing this for distribution is just too much of a hassle to even consider using docker. I understand the logic behind static dockerfiles but I am personally not satisfied with asking people to install third party dependencies (docker-compose) for this.

[wontonst]

+1

We have different versions of an image myimage:myversion

The version is tagged to each revision of our software, so when we do an arbitrary checkout, the correct Docker image is generated. There are a lot of moving parts in our system that makes a hard coded Dockerfile undesirable.

I don't see any reason why

ARG version
FROM myimage:$version

would be of any harm. If you don't like it, then in your organization make it a rule that ARG cannot come before FROM, but for the rest of us who would benefit, this should be a feature.

[GordonTheTurtle]

_USER POLL_

_The best way to get notified of updates is to use the _Subscribe_ button on this page._

Please don't use "+1" or "I have this too" comments on issues. We automatically
collect those comments to keep the thread short.

The people listed below have upvoted this issue by leaving a +1 comment:

@junneyang
@dflock

[bamarni]

There are also other common scenarios than a dynamic base tag, like dynamic bloc inclusion, looking at the official php dockerfiles for example (https://github.com/docker-library/php).

I'm using a small cli wrapper on top of golang's template package (https://github.com/bamarni/dockplate), maybe some of you will find it useful too.

[danielcaldwell]

This is exactly the functionality I need.

I use docker-compose to build a stack of images based off of different branches in the source tree. When a branch is built, the images are specific to that branch of code. Then the regression tests are built against that set of images.

Being able to tag the images with the branch and then reference the specific branched images is precisely what I need to be able to set up my continuous integration pipeline.

In docker compose you see this:

services: 
    myservice: 
         image: myserver_${BRANCH_NAME}

then in the docker file:

FROM myserver_${BRANCH_NAME}

This would make it possible to run multiple branches of the software independent of each other and test them as well. Then once the tests are successful, and if it's the release branch, we push the images to the repository for deploying on the production server.

[aacebedo]

To solve this issue I am currently using Rocker (github.com/grammarly/rocker)

[emk]

This also comes up frequently with continuous integration systems. We have two GoCD pipelines:

• base: This builds example/base and tags it with a build number. This build number is automatically provided to later pipeline stages as $GO_DEPENDENCY_LABEL_BASE.
• app: This takes the example/base version specified in $GO_DEPENDENCY_LABEL_BASE, and uses it to build example/app.

So when we're making official builds, we want:

FROM example/base:$GO_DEPENDENCY_LABEL_BASE

When we're developing locally, it's safe to use:

FROM example/base:latest

One obvious way to implement this would be:

ARG GO_DEPENDENCY_LABEL_BASE=latest
FROM example/base:${GO_DEPENDENCY_LABEL_BASE}

We could then invoke docker build with --build-arg GO_DEPENDENCY_LABEL_BASE=$GO_DEPENDENCY_LABEL_BASE when running under GoCD, allowing us to lock to specified base image.

[kontinuity]

I ended up using Rocker to solve the dynamic Dockerfile problem. https://github.com/grammarly/rocker

[dnephin]

Implemented in #31352, please discuss

[erikbgithub]

Would this be the recommended solution for different architectures, e.g. using ppc64le/debian:latest (PowerPC) instead of debian:latest (x86)?

[thaJeztah]

@erikbgithub definitely an option if the rest of the dockerfile is exactly the same. It all depends on your use-case, so difficult to "recommend" (or not) what works best for you

[erikbgithub]

@thaJeztah I'm a smart, flexible developer. I can adapt my Dockerfiles to the recommended way, promised. "The same" should not be the docker file but the user experience. In theory the best would be to docker pull debian:latest and have docker figure out the architecture and different image itself.

[thaJeztah]

In theory the best would be to docker pull debian:latest and have docker figure out the architecture and different image itself.

The infrastructure for that is in place, and it's already possible (multi-arch images), but there's still some changes to be made (also see https://github.com/moby/moby/pull/27455)

[erikbgithub]

@thaJeztah Yes, that's exactly what I wanted to know. Thanks. So what I've read earlier in a blog post is not correct. It's not out there yet, but just these days in active development.

[thaJeztah]

The feature itself "is out there" - you _can_ already publish a multi-arch image on Docker Hub / a Docker registry, but some changes in the format may still be worked on, and therefore it's not yet used for the official images.

[montanaflynn]

This is now possible if anyone comes here looking for answers:

https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact

[nmDejamobile]

Cool feature. I use it in Gitlab CI context to pass the android sdk version we built an image for on the first sto to the second build that makes an image with fastlane and so I can target my FROM to the previously generated image. Then I tag it with this version number and latest.

[Arjunarus]

Hi all.
I use Docker 18.09, maybe it already fixed in 19.03, I do not able to test it now.
But in my case this minimal Dockerfile doesn't works:
ARG BASE_REPO
ARG BASE_TAG
FROM ${BASE_REPO}:${BASE_TAG}
ARG PARENT_REPO
ARG PARENT_TAG
FROM ${PARENT_REPO}:${PARENT_TAG}

On the second FROM getting error "invalid reference format". Its important to have different values for BASE_* and PARENT_* arguments. It's OK if the same.
I use this command for build, but could be any different images:
docker build --build-arg BASE_REPO=ubuntu --build-arg BASE_TAG=18.04 --build-arg PARENT_REPO=centos --build-arg PARENT_TAG=7 -t test:TEST e:\path_to_test

I discover that to make it works you need to move PARENT_* args before the first FROM, like this:
ARG BASE_REPO
ARG BASE_TAG
ARG PARENT_REPO
ARG PARENT_TAG
FROM ${BASE_REPO}:${BASE_TAG}
FROM ${PARENT_REPO}:${PARENT_TAG}
it looks strange and not follow to documentation:
"An ARG declared before a FROM is outside of a build stage, so it can’t be used in any instruction after a FROM. To use the default value of an ARG declared before the first FROM use an ARG instruction without a value inside of a build stage"

[thaJeztah]

This is because of the scope of ARGs;

• ARGs are scoped per stage. Other stages do not have access to them, and stages begin after each FROM line, and end before the next FROM (if any).
• An ARG that's declared _before_ the first FROM is not within any stage, and thus a "global" ARG.
• Given that the FROM line for each stage is in the "global" stage, those lines have access to the global ARGs, but don't have access to ARGs that are defined in other stages (which could be the stage "above" it)
• An ARG that's declared inside a stage, but _does not have a value specified_ can inherit its value from a global ARG with the same name

# These `ARG`s are in the `"global" scope (before first `FROM` in the Dockerfile)
ARG GLOBALARG_ONE=globalarg_one_default_value
ARG GLOBALARG_TWO=globalarg_two_default_value

# `FROM` lines itself are in the "global" scope, and therefore are granted
# access to the global `ARG`s
FROM myimage:${GLOBALARG_ONE} AS stage1
# stage1 starts here (after the `FROM`)_________________________________________

# Declare ARGs for stage1
ARG STAGE1_ARG=foo

# Declare ARG for stage1. If no value is specified, this grants permission to
# use the value from the globally defined ARG with the same name
ARG GLOBALARG_ONE

# If a default value is set here, this value will be used, unless overridden
# on the command-line (`--build-arg GLOBALARG_TWO=override`)
ARG GLOBALARG_TWO=my-local-value

# stage1 ends here (before the next `FROM`)_____________________________________

# `FROM` lines itself are in the "global" scope, and therefore are granted
# access to the global `ARG`s
FROM myimage:${GLOBALARG_ONE} AS stage1
# stage2 starts here (after the `FROM`)_________________________________________

# Declare ARGs for stage2
ARG STAGE2_ARG=bar

# stage2 ends here (before the next `FROM` or end of the Dockerfile)_____________

[Arjunarus]

Thanks for the answer. Now its clear.
Does the official documentation have the description about this global scope of the ARGs and FROM somewhere? I read but didn't get it.

[thaJeztah]

I _think_ there was a section written about it, but it's definitely something that could be improved in the docs

[TBBle]

https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact mentions this restriction under the FROM reference (now, not sure it did two months ago, perhaps that was updated based on @thaJeztah's example)...

FROM instructions support variables that are declared by any ARG instructions that occur before the first FROM.
[... example elided]
An ARG declared before a FROM is outside of a build stage, so it can’t be used in any instruction after a FROM. To use the default value of an ARG declared before the first FROM use an ARG instruction without a value inside of a build stage:
[... example elided]

[thaJeztah]

Not sure it did two months ago, perhaps that was updated based on @thaJeztah's example

Looks like that was added in https://github.com/moby/moby/pull/32486 (2017)