Compose: Proposal: Don't build images

[dnephin]

An even more aggressive step from #693, compose could entirely remove the ability to build images.

Rational: If compose is a tool for composing services, the process of building images for those services is distinct from the concerns of running services.

The docker images already provides a great interface for separating these concerns. The existing compose support for building an image is pretty primitive, and could be easily replaced.

There are many different build scenarios, each of which could be handled by a separate tool:

• build from urls (#1209, #1369)
• the naming of images (#1363, #2092)
• using builder containers (#1257)
• pushing images to a registry (#1126)
• tagging images (#974, #213)
• don't build the same path twice (#963)
• pulling base images (#726)
• build ordering (#295, #610, #583, #663, #1548)
• building base images (https://github.com/docker/compose/issues/1661#issuecomment-146724163)
• --build-arg command line flag (https://github.com/docker/compose/issues/2111#issuecomment-174247444)

Some of these problems are already solved by existing build tools:

• https://github.com/6si/shipwright - building many containers from a git repo, where images extend each other
• https://github.com/openshift/source-to-image - creating minimal images using a builder container
  (There are probably other tools that I'm not aware of at this time.)

Backwards Compatibility

In order to preserve some form of backwards compatibility we could:

• in one release, warn if a build tag exists, and continue to build images
• in the next release
  
  
  
  • allow both build and image keys for a service
  
  
  • ignore build and dockerfile keys
  
  
  • provide a new command to build images in the way that compose does now (based on the same config)
  
  
• finally in some future release remove the build and dockerfile fields, and require a separate config for building images. Initially that config might be in a similar format, but likely it would evolve to support many of the scenarios described above

[kojiromike]

I like the UNIX thinking behind this proposal, but I'm having a hard time envisioning it as anything other than a burden without some examples of how it would work.

1. Compose files are very static (even with extends). Injecting dynamic image names is a chore. Worse if you have many dynamic images.
2. Compose doesn't currently accept image ids. If you don't use build, you have to name the image. But it's not always appropriate to name an image at all.

[dnephin]

I agree that supporting image names and tags from environment variables would have to be supported before we were to do anything like this.

Compose does accept image ids (it was broken for a release, but I think the fix is in 1.3.0). I don't think that naming an image is a huge burden, but with environment variable support, using image ids should work as well.

[aanand]

It's definitely true that building images is complicated, that Compose's primary job is orchestration and not building, and that the Docker image is the natural place to draw that line.

Swarm has already drawn that line by not supporting building at all, and down the line it's likely that building will be moved out of even the Docker daemon, to become a client-side process which just performs create/start/commit calls against the API.

Still - alongside everything else, Compose remains a tool for development environments, and having a simplistic build option serves the 80% "git clone, docker-compose up" case very well, so I'm reluctant to let go of it. It might be worth trying this docs-first, rewriting one or two of the tutorials without build and see how they look - but even then, we're examining ultra-simple one-build scenarios.

In any case, when it comes to the multitude of use cases you've linked to where Compose falls short, we might do well to recommend tools or approaches in the documentation (I suspect a Makefile would serve a lot of them).

[prologic]

I kind of don't mind having build removed either. To be honest I don't use it all that much. I typically use docker-compose to "compose" services and really use it as my "deployment" tool.

[thaJeztah]

I'm still on the fence on this one. On one hand, it makes sense to make docker-compose only "orchestrate". On the other hand, having a single configuration file that defines which Dockerfiles have to be built for each service and which context to use for them, is really convenient.

Basically, it allows one to describe the whole "stack" in a single file, and just looking at the file describes what the project looks like.

[prologic]

I guess it is nice and convenient to do (_in development_) dc up (_my alias for docker-compose_) which used ot be fig up with a bunch of build: ... stanzas.

Problem is from a philosophical point of view; I don't really find myself doing this that often and could live with splitting this "workflow" out to a searpate "integration" and "testing" phase where I can build up the whole "stack"; but the parts are built individually.

[thaJeztah]

Perhaps docker-compose should have some "hooks" that can trigger external tooling. (Really thinking out loud here)

[prologic]

TBQH though; I am kind of torn over this. Whilst I don't use the build feature of docker-compose myself that much it can still be nonetheless useful in some situations and for others :)

[funkyfuture]

i like the idea of dropping the build-, and consequently the pull-command. in order to make these a convenience in every situation would require a blow-up of the configuration options.

Perhaps docker-compose should have some "hooks" that can trigger external tooling. (Really thinking out loud here)

word. just let users place an executable _pre-up in a project-dir and let 'em bear the responsibility for what's goin' on there.

[kojiromike]

just let users place an executable _pre-up in a project-dir

Environment variables, please! :)

COMPOSE_PRE_UP_HOOK?

[funkyfuture]

that's what i mean. scripts can access environment and write env-files or manipulate a project-configuration.

[diginc]

I use build and Dockerfiles which are FROM the public registry to handle customizing package sources and proxy stuffs, it is useful.

[josephpage]

From its origin, fig was a tool for development environnement, and still today it's the only (and the best !) CLI tool for that purpose.

Because I uses docker-compose intensively for development, I don't agree with @dnephin that compose is a only tool for composing services.
Kubernetes is for composing services in QA/production environnement and managing them.
But docker-compose is for launching lightweight stacks for dev/QA or even non-critical production quickly.

In my dreams, only git clone + docker-compose up [-d] should be required to start coding and live-testing a project.

Building an image is one of the basic steps to learn Docker
Building an image is the first step of developing with Docker.
Building an image is one essential part of the fig up's magic !

[kojiromike]

I essentially agree with @josephpage on this: If a tool like docker-compose is not going to be able to build directly anymore, then another tool must fill this gap – I need to be able to deliver a development environment with my code and have a single command for developers to get going on it. I have been able to do that with fig and compose, but am concerned about losing that ability.

[etc0de]

I entirely rely on "docker build" finding out what needs to be built from my docker-compose.yml file, so if that was removed that would greatly disrupt my workflow.

[dnephin]

Another argument for removing build and dockerfile is that it's quite confusing for new developers coming to docker and compose (#1901 is just one example, but there are a few every week).

Have a config where 2 fields apply to one operation, and the other 20+ fields apply to a completely different operation isn't very intuitive.

[etc0de]

Again I would highly suggest just removing the implicit build triggered by docker-compose up instead which is the main source of confusion. (also the issue you linked doesn't seem to be specific to docker-compose's build.)

Docker compose is right now a nice tool to manage all around my docker image groups which pretty much covers everything needed. While the build support is primitive, it still saves time and makes it easier to manage the whole thing including the necessary build step in one simple tool.

One important point why docker-compose should cover everything and not just the running non-build functionality is that the docker-compose.yml contains information how the images will be _named_ as well. If you remove basic functionality from docker compose to limit its scope, you will force people to parse docker-compose.yml manually with other tools to get that information, or force people to have yet another configuration file lying around which holds it instead. That seems like a nonsensical idea to me.

[kojiromike]

:+1: to @JonasT 's idea.

[jayfk]

:-1:

[michaeljs1990]

I think you will see a hard fork of this project if this happens as it completely destroys the use of it in development.

[dnephin]

I think it's clear that until we have another tool that would replace the build configuration we won't be removing build from compose. I think it's still important to remember that building images isn't the primary focus of compose, so not every feature request for building images in different ways is going to be supported.

[etc0de]

I think it's still important to remember that building images isn't the primary focus of compose, so not every feature request for building images in different ways is going to be supported.

While that makes a lot of sense, this isn't much of a feature request but more a suggestion for a simple trivial change for a user experience problem. Lack of features is very reasonable if something isn't your main area, but that doesn't mean the ones that are present couldn't be optimized a bit when the user experience is too confusing.

However, as I stated above I think removing build isn't such a good idea, so I don't really mind this being closed. But I suggest to consider making up not implicitly build the container for a more coherent experience.

(unless this was already solved by some other change, e.g. building with every up or something similar)