Compose: Define services which are not started by default

[bfirsh]

Users quite often define maintenance scripts, test suites, debugging systems in their Compose files which they don't want to run when they do docker-compose up.

There should be some way of defining what services are started by default, but can be still be run manually by doing docker-compose up servicename or docker-compose run servicename ....

Possible solutions

1) Recommend users to use a separate Compose file
2) Add an option to services to make them not start by default
3) Add a top-level configuration option to define the default services
4) Add a concept of a thing like a service, but is just for one-off commands ("scripts", "tasks", etc...)

(Please suggest others if you have ideas.)

Data points:

• https://github.com/docker/compose/issues/697
• https://github.com/docker/compose/issues/912
• https://github.com/docker/compose/issues/942
• https://github.com/docker/compose/issues/1439
• https://github.com/docker/compose/issues/1547
• #542
• "test" service in https://github.com/heroku/logplex/blob/master/docker-compose.yml
• @cpuguy83: "I've got a little helper service in my compose yaml that injects a bunch of stuff into redis for hipache, which of course exits when it is done. Can't use compose up w/o the -d" this is fixed now that we don't exit from compose up until all services have exited.

[dnephin]

+1 for option 1

I think adding services that aren't actually part of a composition, but just happen to need to link/attach to it, is a bad design choice. There are a few other use cases that would be solved by allowing some form of includes syntax. Solving multiple problems with a single feature is always nice.

Some of these issues (the ones that deal with data-only containers, #942, the last comment from @cpuguy83) are actually already fixed by #1754, I don't think we need to consider them a problem any more (after 1.5).

[kojiromike]

We develop Magento extensions. To do that, we need a simple way to run a webstore on a LAMP stack. Compose makes that easy. But we also want to run phpunit, various static analysis tools, documentation builders, etc. In fact, the vast majority of our "services" are just commands, like docker-compose run --rm phplint. Ideally, an unqualified command like

docker-compose up -d

would only start long-running service (actual _services_ as it were), and would not trigger other effects like phpunit, unnecessarily. (It's also important that it not trigger things out-of-order, like running Selenium tests before the web service is up.)

All we are really trying to do is take the pain of environment management away from developers, not run production services. I don't know if @dnephin 's comment is representative of the direction Compose is headed or not, but it seems ambiguous to me whether Compose is still planning to serve the niche Fig started out in. (Because, to be clear, I think option 1 does not support the ease-of-use that developers need to use this as Fig apparently intended.) I totally understand if Compose doesn't want to have multiple responsibilities, but I hope that someone can let us know, clearly, if those of us using this product for fast, isolated development environments should head in another direction.

[ryneeverett]

@kojiromike, what is it about (1) that is unsuitable for your use case? Is it just about the command semantics (docker-compose run --rm phplint vs. docker-compose --file phplint.yml up)?

[ppg]

Its that docker-compose up -d tries to 'start' phplint and docker-compose ps reports that that the phplint 'service' is down. In reality that's not a service container its a tool container, it should have no concept of up/down. It was my understanding that tool containers are embraced by docker (its how you would run something like redis-cli, clearly not a 'service') and while I use them more in development, I think they have a place for production too; like why have say awk installed on production machines or in containers, why not run it through a container with linking to get predictable behavior out of it.

[chrisguidry]

:+1: I routinely find myself wanting to create a tests container alongside my other services to encapsulate running unit tests. My "workaround" is to set the command to "/bin/true" and run the container with the unit test command specified on the CLI. Being able to specify which containers should start at up time would be great!

(btw, nice work all around, folks)

cc @jameydeorio

[kojiromike]

@ryneeverett The semantics are one part of it. It's a problem of self-documenting and findability. Currently, we tell developers docker-compose run --rm foo bar. We invite them to create a shell function or alias, but we don't maintain a standard alias/rcfile for projects. (We don't want to standardize things outside of containers; we want to use Docker _to_ standardize.) Adding a new file for some commands creates a hierarchy of importance: The docker-compose.yml file becomes the "default" file for important things, and the "other" file becomes less important.

The other thing is just maintaining relationships between services becomes more onerous. Just because we want something not to run by default doesn't mean it doesn't use a link or volume from a service that is long-running. extends doesn't actually provide all the facilities we would need to link services to "commands" (one-run services). Even if it did, if we have to use multiple yaml files, we will be forced to use extends where we otherwise wouldn't need to.

[ryneeverett]

extends doesn't actually provide all the facilities we would need to link services to "commands" (one-run services). Even if it did, if we have to use multiple yaml files, we will be forced to use extends where we otherwise wouldn't need to.

@kojiromike That's what I suspected. I wonder if you would be satisfied with improved extends support + some way to do subcommands (which would be functionally identical to extends) within a single docker-compose.yml. But perhaps the latter is the same as (4).

[tomfotherby]

2) Add an option to services to make them not start by default

I vote for Option 2. e.g. something like a start: false directive. The advantages are that we avoid the need for multiple compose.yml files or extra config files, you can just read one compose.yml to get the feel of the whole app stack.

[dnephin]

I think our proposed solution for https://github.com/docker/compose/issues/1987#issuecomment-139632238 will handle this. "admin" services can be defined into a separate config, and added with -f when an admin command needs to be run against a composition.

[sherter]

@dnephin The solution in #1987(comment) _does_ handle "admin services", but it _doesn't_ handle "data only containers", right? You would still have to use the command: /bin/true workaround.

[cpuguy83]

You shouldn't need data-only containers with compose as compose will handle swapping the volumes to re-created containers for you.

[sherter]

@cpuguy83 In the following example the data container is not really necessary, but if I want to change the volume, I only need to look at one designated place.

nginx:
  image: nginx:1.9
  volumes_from:
  - data

php:
  image: php:5.6-fpm
  volumes_from:
  - data

data:
  image: debian:jessie
  volumes:
  - ./:/app

However, I agree that this is a corner case and adding additional syntax is probably not worth it. I'm just saying that the proposed solution doesn't handle this case.

[dnephin]

True, some command is still required for data volumes, so you would have to use a minimal image to support that command.

I'm not completely up-to-date with the new volumes API that's coming out, but my hope is that we will be able to add a volumes: section to compose which will handle the data volume in a better way (instead of requiring a container for them).

[CWSpear]

This doesn't necessarily make it the right choice, but I think learning curve/ease of understanding should be considered.

I feel like (2) is the easiest to comprehend. I don't really have a way to confirm this, but my gut says most people who aren't intimately familiar with all of docker-compose's options that run into the problem we're trying to solve here say, "I wish there was some way to get that container to _not start_ when I run docker-compose up," and they see start: false and bam, we're done and happy.

They don't say, "If only there was a way I could create a 2nd file with an awkward linking story to solve this..." (tho I realize that https://github.com/docker/compose/issues/1987#issuecomment-139632238 helps with the "awkward linking story," ya?).

(4) was kind of vague, but a dedicated avenue for the way for scripts and one-offs like this fits this "makes sense" bill.

[kalbasit]

Today I was looking for exactly (2) but ended up with my least favorite solution, a second YML file. My use case:

I have couple of container and all of them link to the same mongo container. I'd like to offer myself and the team the ability to load fixtures into the mongo database and I figured the easiest way to do that is to load the mongo container under a different name fixtures that itself link to mongo and then run mongorestore --host mongo --port 27017 --drop /dump. Since we don't want to load the fixtures at all time, it felt natural to have it with a start: false but ended up having a separate YML file for both containers fixtures and mongo.

Works well but start: false is much cleaner IMO. If I would have 10 or more permutations of this so-called fixtures container then yes start: false would be a bad idea and I'd go with (1).

[tonivdv]

Just had to make a compose file where some services should not be executed with the docker-compose up -d command. For me option (2) would be the best solution to my problem.

[mgriego]

I've run across this as well. Not sure which I think is the best solution, but it what it comes down to for me is that I need the ability to build images that are utilized as part of a composition. They're for ephemeral containers, but I need the image to be ready for use so I can run the container on demand.

[qcho]

Hi! same issue here, glad there is a nice grouping issue, great work!

In my case I have a python stack working with a couple of containers with services. Everything is working fine, but I've just found a case out of the stack. I'm managing static dependencies with bower, and want to run the bower command inside a container, it's just a script that we run once a while.

Would be great to run this script within the compose structure we have. I imagen something like:
docker-compose run node bower install

So I like the 2th of 4th option. Just need the service not to start, it's not needed :P

If there is some consensus I could try to send a pull-req for something like "restart"
start: always
maybe... dunno.

[jmreicha]

Another vote for option 2.

[gittycat]

I would prefer option 2 as well. The new Docker Data Containers can be specified in a docker-compose file but they don't need to be running to be used.

[snussbaumer]

Option 2 for me too

[philcal]

Option 2 would be great.

[IAmJulianAcosta]

+1 for option 2

[xaka]

+1 for option 2. Maintaining multiple files and typing their names is just wasting of time. One single boolean flag would rule them all. :beers:

[pmoust]

+1 for opt 4,
If I must pick a second favourite its opt 2.

[alexsapran]

True that the best options is 2,4 if i can +1 both then thats what i do if not I am going with the majority and vote for 2.

[aris-b]

Voting for 2

[stafot]

both 2 and 4 are nice to have.
+1 for 2

[sherter]

I don't think it's necessary to introduce a new concept for tasks. These read very well in my opinion and are possible without many modifications:

// 'task' being a service that manages all my tasks, probably with some custom entrypoint script
docker-compose run task tests

// 'tests' being a service specifically for testing with the default command set to run my tests
docker-compose run tests

The current "solution" with the separate compose file is not really a solution in my opinion. As @xaka said, nobody wants to type all this:

docker-compose -f default-file.yml -f additional-tasks-file.yml  run task myTask

What you'll end up with is a ./run-task script that adds all the boilerplate before myTask for you. But now you have added another entrypoint and interface to your multi-container-app. Developers see docker-compose.yml and think: _"Oh great, a compose app. I know how to handle this thing!"_ And now you have to tell them: _"Right, you can manage it with docker-compose like every other compose app... oh, but wait... there is also this additional script you need to know..."_

start: false / up: false / some-additional-flag-to-a-service: false is probably the simplest thing to implement, but most likely also the clearest and easiest to understand. And it would improve usability so much.

[mgriego]

what @sherter said. :arrow_up:

[ThiefMaster]

:+1: for that. separate dockerfile is a huge pain especially when networks are involved

[gittycat]

The start: true|false approach is too limited. What if some services are used for testing, some others for admin and the rest for normal operation?

I would prefer adding the notion of grouping to services.

    myservice:
       group: `admin`

If the group attribute is not defined, default is assumed.
That way we could start the default and admin services using docker-compose up -g admin -d.

Better yet, make groups an array.

[kojiromike]

Creating group-classes of services seems like it would be a powerful feature, but it also seems tangential to this issue.

[qcho]

On docker-compose version 2, every container is declared inside the services: top-level-item.
Declaring a service, with start: never to run a script sounds wrong.
So considering the new format, shouldn't we declare an extra top-level item apart from _services_, _volumes_ and _networks_?

My proposal:

• add a new scripts:, transients: (or think a better name) top level item to v2.
• inside transients you can define a container as you would do with any service
• the only difference is that they are not going to _start by default_, since they are intended for transient use only.

Example:

version: "2"

services:
  web:
    image: myapp
    networks:
      - front
      - back
    volumes:
      - /usr/src/app/
  redis:
    image: redis
    volumes:
      - redis-data:/var/lib/redis
    networks:
      - back

scripts:
  bower:
    image: my_image_with_bower
    volumes_from: web
    working_dir: /usr/src/app/static
    command: "bower"
// maybe would be great to place something like "bower $@"
// to define where you want the cli arguments to be placed, by default at the end.

volumes:
  redis-data:
    driver: flocker

networks:
  front:
    driver: overlay
  back:
    driver: overlay

And then you could run:

docker-compose script bower <EXTRA_CMD_ARGUMENTS |  default nothing>

docker-compose script bower install

Concerns:

• It sounds OverPowered to create another top level item just for this purpose of managing start, but it's derived by the fact that the new format is declaring the behaviour in the top level.
• I'm trying to imagine extra flexibility (and problems) this may add.

Lastly, on the _groups feature_, sounds nice but if you have that many grouping I don't see the problem on creating docker-compose files for each one of them. Probably the feature that you want is docker-compose file inheritance, that would be awesome!

PD: @bfirsh maybe if you like the idea you can add it to the "Possible Suggestions"

On second though this is a revamp of the 4th suggestion because of the imminent new format declaring services.

[kwiky]

Voting for option 2 : Clearest, easiest to read, and don't need to learn new concepts
Just want a service that don't start by default

[cpuguy83]

What about something like disabled: true

[kojiromike]

@cpuguy83 That would seem to imply the whole service is disabled, even for run. I'd find it confusing.

@qcho hmm, now that I've refamiliarized myself with Docker since 1.6, I can see what you and @gittycat are talking about. In that sense, I really like @gittycat 's approach. I picture (blue sky) an interface like:

groups: # if no groups, all services are in the "default" group by…default
    - foo
    - bar
services:
  foo:
    image: imagine
    groups: foo
  bar:
    image: energy
    groups: bar
  baz:
    image: money
    # no groups, so "default"
   quux:
    image: word
    groups:
      - bar
      - default

Meanwhile, in the shell…

docker-compose up -d # Up all services in default group, so 'baz' and quux
docker-compose up -d --groups foo # Up all services in the foo group
docker-compose up -d --groups foo,default # You get the idea
docker-compose run --rm bar somecommand # We never started 'bar', so it can be a one-off command

An approach like this would be awesome and obviate the need for this ticket, but does go beyond its scope.

[cpuguy83]

@kojiromike I don't think so. This is how init systems refer to services that shouldn't start up automatically.

[cpuguy83]

It's also simple and any "confusion" can be resolved with documentation.
I also find it much less confusing than this grouping, which is completely unrelated to the concept of a service starting.

[kojiromike]

@cpuguy83 I think the semantics of "services" is exactly what is confusing in the first place. I agree that the grouping thing is scope creep. In that sense I prefer option 4/@qcho 's approach, where they clearly differentiate "services" from "things that are not services".

[qcho]

Thats the point, why should I put a container that will never run a service under a "service...disabled" category. Sounds like an ugly patch someone made and It's not intuitive.

Maybe the "version 2" of the format should have taken this issue into account. For example another spec could be

version: 3?
containers:
  webserver:
    ...
  database:
    ...
  cache:
    ...
  some_script_container:

services: (they are groups):
  production:
    webserver:
      ...
    database:
      ...
    cache:
      ...
  development:
    webserver:
      ... DEFINE SOME CUSTOM DEV STUFF ABOVE basic container definition
    database:
      ...
    cache:
      ...     

Ok now we have proper services, with groups definition. I can start production or development service group. Or just run a script in the some_script_container. since it's not defined in any services no one will be started

[cpuguy83]

@qcho Depends on the definition of a service, but in any case -- I'd call into question something which should only be run on occasion, and the only a human can determine if it should be run or not.

So let's say compose adopts something like a job object.

1. jobs only get run once
2. jobs are started after services
3. jobs are expected to exit
4. ???

Now compose also needs to keep some local state about the job, which it currently doesn't do at all.

[mgriego]

Which is why adding a simple auto-up: false or autorun: false is the easiest and least creepy (scopewise) way to handle this.

[qcho]

@cpuguy83 I think you are extending what I'm saying with something more complex. I don't intend docker-compose knowing about any job or workflow. External scripts can be made for that. What I intend is that docker-compose should define the entire stack for an application to run.

I can Imagine myself creating a script and keeping the state you are saying. Even calling the script before service start, not just after. What I don't imagine is myself parsing the composer file because I need to check what volumes I need to import, or what configuration i need to extract from it on an unknown container to run some script that applies to it. So I think docker-compose should give me that container; running the script and keeping state is my problem.

I don't see why people keep saying that

services:
  my-script-container:
    auto-up:false

Is simplier than:

scripts/containers/transients/you_name_it:
  my-script-container:

It's the same level of complexity. But less hacky semantically.

[jimzucker]

To get the ideas on one thread, reference #2803

Use case: You have a project with many components and users what to pick and chose which to install but he compose file installs them all.

Proposal: We add a option to put int he docker-cmpose.overrider.yml to exclude a image defined in the docker.compose

ie

some-image:
exclude: yes

The behavior would be to ignore that entry in the docker-compose.yml

Can you let me know what the team thinks, I would be in interested in making the change.

[KiraTsunayoshi]

Voting for #2 as well...just ran into this need today.

@jimzucker's proposal would also work, though I like the thought of seeing "start: false" in the main .yaml file to immediately clue you in that this 'service' won't run unless you call it explicitly. Otherwise you (or in my case the end-user/developer you handed the docker-compose files to) need to remember to look for an override file.

[LLFourn]

+1 for 2. I have a situation where I need to build a docker image as part of compose but it doesn't run as a service. The other services (which have docker sock mounted inside them) run containers from the image every so often.

[niko]

+1 for 2. And +1 for "auto-up" or "auto-run" as wording.

And can't the cases for groups of services (as explained by @gittycat) be handled via environment variables ala "auto-up: ${ADMIN}"?

[acran]

I also see real use cases to mark services in the docker-compose.yml to be not automatically started with a simple docker-compose up but instead to be started only explicitly.

Solution 1) is one possible way now but in my opinion too cumbersome since one has to specify one or more yml files instead of just calling docker-compose up and having to split or even duplicate the files.

Since I'd really like to see something like solution 2) (as many others do too) I implemented a proof-of-concept of this in #3047 so you can play around a bit with it to see whether it could be a viable solution.

[xcash]

+1 for option 2

[pvgoddijn]

+1 for option 2

[antonypp]

+1 for option 2

[csaftoiu]

+1 for both options 2 and 4

[hugochinchilla]

+1 for options 2 and 4

[mathroc]

+1 for option 4. some configuration should not be allowed for scripts (eg: restart: always)

with option 2, it could have this weird case:

service:
  run_tests:
    build: ./tests/
    restart: always
    auto-start: "false"

what would that mean ?

[niko]

@mathroc: It means: "Don't auto-start this container when compose is executed. When this container stops (after being started explicitly), restart regardless of exit code." What's weird about that?

[mathroc]

@niko oh right, I should have a thought about that a bit longer before posting...

changing my "vote" to +1 for option 2

[donnex]

+1 for option 2, I've needed this in multiple projects.

[acran]

pinging @bfirsh @dnephin:
Could you give an update regarding this issue? Since most comments here are in favor of option 2, are there currently any plans to implement something like that (or option 3/4)?
I could polish up my pull request (#3047) and complete it with tests and documentation if you'd consider merging it then.

Thank you.

[alexbde]

+1 for option 2

[atilaromero]

+1 for option 2

[brettmcintosh]

+1 for option 2

[rawkode]

I think that option 1 could work, but we need a better way to reduce the verboseness of:

docker-compose -f docker-compose.yml -f docker-compose.tests.yml up

Perhaps would could add a simplified / augment flag? docker-compose --augment tests up and automatically resolve if it's a variation of docker-compose.yml - otherwise the user would need to be explicit.

I really like the idea of a new top level keyword though, perhaps suites and commands ?

The following configuration would allow for:

docker-compose run -s application
docker-compose run -c cache-clear

suites:
    application:
        services:
            - database
            - memcached
            - application
    tests:
        extends: application
        services:
            - application

commands:
    cache-clear:
        service: application
        workdir: /var/www
        command/entrypoint: php app/console ca:cl

[docteurklein]

I can see in the code that there is already the notion of "oneoff" services (IMHO only used when docker-compose run is called.
Could it be re-used?
If I labelize a container with com.docker.compose.oneoff=True, would it try to start it?

Otherwise, I vote for option 2.

[ankon]

I just stumbled over this, so here my two use cases:

1. "global" environment variables: The suggested approach here is to use extends, which I'm fine with ... but now I need to define an image for that thing I'm extending from, and cannot even use scratch (so ended up with https://hub.docker.com/r/tianon/true/).
2. "scaling" services: by default I have one MongoDB instance, but for some tests I'd like to define a second inactive one, which I can bring up for those tests. Here using extends in a different file looks like a viable idea (still learning about docker/docker-compose and the best practices involved).

[originalgremlin]

+1 for option 2

[grachevko]

+1 for option 2

[jcsalem]

+1 for option 2
Seems like a simple change...

[jkirkby91]

+1 option 2,

in the use case of multiple containers sharing a common set of envs and configs it makes sense to build a base container to extend from but not to actually ever start the container

auto-start:true|false

[julian7]

Having a nice solution since last September called rocker-compose, I'd recommend run: _always_ | _once_ | _never_. I also like state: _running_ | _ran_ | _created_ too.

[jkirkby91]

not a fan of using a separate project just for this feature, i ended up stealing the True asm code command out of tianon/true project until this is implemented

[gimoh]

I also initially thought option 2 would be best but now I'm starting to think it would be too restrictive and soon enough after adding it, a need for run-once tasks would pop up. In fact I'm working on a project now where I have a need for the following use cases:

• within the stack I want to be able to define containers that run once on initial deployment and any future runs are only triggered externally (e.g. schedule/event, via API call, etc.) so that they can initialize/update state of main service container within the same stack (e.g. DB data/schema, populating data volumes, etc.)
• within the stack I want to be able to define containers that are only triggered externally so that they can be used to encapsulate utility/admin tasks (e.g. state/data inspection, etc.) and processes that are triggered on schedule (e.g. backup) or by external events (e.g. trigger a process on changes to an S3 bucket)

In my case the distinction between state: created (ran 0 times) and state: ran (ran >= 1 times) is important as some of the utility/admin tasks may be destructive and are only to be used in certain circumstances like migrating services.

Given that I'm now leaning more towards state: running | ran | created as in rocker-compose or option 4 with top level _task_ or _job_ object + ability to express a dependency so that a service can trigger a task to run before/after itself.

[thenewguy]

I would like to mention this is important for my use case. I use docker-compose to create a testing environment and run a test suite. The problem with starting all containers at once is that I cannot easily allow the service containers (like database or celery workers) to initialize before running the test suite.

It would be preferable to be able to accomplish something like:

$ docker-compose --file="docker-compose-testing.yml" up -d --exclude=tests
$ sleep 5
$ docker-compose --file="docker-compose-testing.yml" run --rm tests

You can imagine something more elaborate than sleep 5 could be written to verify initialization has occurred but it isn't necessary for the example snippet.

I did not read an option about a cli flag. So I would propose an --exclude flag is added to the list of options. The --exclude flag would tell the up command not to start the specified container. The --exclude flag would be parsed such that multiple containers can be excluded from start up as necessary.

[SvenNederhoff]

:+1:

[ncoghlan]

Weighing in with another use case where option 2 would be a suitable solution: iterating on a new subservice within an existing application. Most developers on the project aren't going to need the new service yet, and you definitely don't want problems with it hindering their work, but you may also want to avoid maintaining an out-of-tree branch for an extended period of time.

A separate "docker-compose-experimental.yml" file _could_ work, but it means all the documentation for working on that component needs to be written one way while the component is still experimental, and then revised once it becomes part of the standard set of services.

[synthead]

+1 to option 2

[jimzucker]

+1 for Option 2

[trilliput]

+1 for option 2

[hrbu]

+1 for option 2

[chkelly]

+1 for option 2

[jimzucker]

@acran I would like to see you submit your pull request to see if we can get some traction moving this forward, maybe if you rebase to eliminate the conflict on #3047 it will get picked up, not sure how to get attention to this.

[acran]

@jimzucker well #3047 already is a pull request and I'm still waiting for some comments on it by the devs :disappointed:

See my comment above: making the PR merge-worthy still requires some polishing, i.e. update related docs and test cases. But I'd rather not do that work without knowing that this approach will be considered to be integrated by upstream. Otherwise my efforts on this could be pointless...

[garcianavalon]

+1 option 2

[dattran-vn01]

@acran, I'd like to help you to update related docs and test cases. Is it ok ?

[christoph-kluge]

+1 option 2

[pacuna]

or maybe define a kind of Job, like in Kubernetes?

[anacicconi]

+1 for option 2

We have a docker-compose with multiple services and one of them (the database) must be launched on the development environment but not on the production environment. Therefore, we have to maintain two docker-compose files. Option 2 is exactly what we are looking for!

[srossross]

:+1: for option 2

[illegalnumbers]

+1 option 2

[jimzucker]

@acran I am reaching out to some connects here in SF to find out the right process/channel to get feedback so this can move forward, stay tuned ;)

[dbpolito]

👍 for option 2

[lulurun]

+1 for option 2

[yiannisfloater]

+1 option 2

[drew-r]

+1 option 2

[sanglt]

Also I'm vote for option 2

[bfirsh]

Another idea: now we have top-level networks and volumes, we could add an images option too. This would be a hint to Compose that these images have to be pulled or built for the application to work, but they wouldn't be run when you do docker-compose up.

This would work for two use-cases, off the top of my head:

1) The common problem here, where you want some kind of management command to run. docker-compose run could fall back to running images if the service doesn't exist.
2) Apps like this which run Docker images but they aren't running all the time.

/cc @dnephin @aanand @shin-

[mgriego]

Except that it would be nice to be able to configure all of the options a short-term container would need (ie the volumes/mount points, networks, links, name, etc) in the docker-compose file so that you don't have to keep adding them every time you need to run that container...

On September 13, 2016 3:58:41 AM CDT, Ben Firshman [email protected] wrote:

Another idea: now we have top-level networks and volumes, we could
add an images option too. This would be a hint to Compose that these
images have to be pulled or built for the application to work, but they
wouldn't be run when you do docker-compose up.

This would work for two use-cases, off the top of my head:

1) The common problem here, where you want some kind of management
command to run. docker-compose run could fall back to running images
if the service doesn't exist.
2) Apps like
this which run
Docker images but they aren't running all the time.

/cc @dnephin @aanand @shin-

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/docker/compose/issues/1896#issuecomment-246618909

Sent from my Android device with K-9 Mail. Please excuse my brevity.

[zxaos]

I agree with @mgriego on this. Think of a build generating container - it'd be best to have the command and options all set up and just docker-compose up foo and have it do the build and exit.

[ankon]

Trying to get more understanding here: Is option 2) just a specialized case of the more general "we want a initial_scale value in docker-compose.yml"? (see #1661, #2496 et. al.)

[mgriego]

@ankon not at all.

[dnephin]

add an images [section]

@bfirsh this starts to move in the direction I took with https://github.com/dnephin/dobi. I moved away from having "sections" (and used just top-level items for each resource instead). I think it would be difficult to make that work with the Compose model.

1. networks and volumes are never really operated on directly, they are just created as needed by a container. Splitting images into their own section isn't going to change the behaviour we have now, it would just make the Compose config more complicated. If you wanted to just build an image, run docker-compose build.
2. I don't think it actually fixes the problems identified in this issue. People want containers, not just the image

[ncoghlan]

Re-reading the thread and the discussion in #2496 about declaring scaling constraints in the compose file, I'm coming around to preferring Option 4 (i.e. a new top-level section) over Option 2.

Specifically, a section name like utilities would be a good fit: components that aren't normally needed, but when you do need them, you want to have preconfigured which other components they're linked to rather than having to get people to construct a suitable docker run incantation themselves.

That approach would then cover not only one-off admin commands, but also auxiliary debugging and introspection services like pgweb and Celery Flower (which are very handy to have running in development, but you typically wouldn't want in production due to the additional security threat surface area they create).

Defining the semantics of these new "not services" also gets a lot easier: they're _exactly_ the same as services (including all future changes to that format), with the sole exception that unqualified commands like docker-compose build, docker-compose pull, and docker-compose up ignore them entirely - if you want any component listed in "utilities" rather than "services" to be processed, you need to name it specifically (although there could perhaps be an --all option, similar to the current one for docker-compose rm). However, the CLI for interacting with these utilities by name would be identical to that for interacting with normal services (unlike the status quo, where you need to specify the extra utility config file), and docker-compose would take care of preventing name conflicts between service and utility definitions.

While the scaling proposal in #2496 could potentially be (ab)used to get this behaviour, it still feels like a workaround rather than a properly designed feature.

[dsem]

I like that design, @ncoghlan.

Instead of utilities what about just naming it .services? This naturally conveys that services and .services use the exact same semantics, the only difference is that .services are not run by default. It's a pretty close analog to filenames that are not displayed by default. I suppose the downside is that the distinction between the two is a bit subtle.

[kojiromike]

I like the basic idea @ncoghlan talks about there, but…

1. Don't agree that _all_ unqualified commands should ignore these utilities by default. In my opinion, all commands should run normally except those that would result in a running container. (So basically, up and start.)
2. Regarding @dsem's comment, I like the semantics of 'utility' better than '.services'. I understand the unixy hidden-file reasoning, but these things aren't merely hidden services, they are service helpers.

[acran]

@bfirsh @ncoghlan I fully agree with @dnephin:

I don't think it actually fixes the problems identified in this issue. People want containers, not just the image

What I - and I think most of the people here - need for my desired workflow are simple services; they're defined like any normal service, they're started/stoped like normal services etc.
The _only_ difference is that they are not automatically started by a simple docker-compose up by default but only when specified explicitly (or pulled in as a dependency).

Therefore I don't see any necessity for a new top-level section or a completely new concept for these one-off containers but only an optional per service config parameter.
I implemented this in my pull request #3047 as a proof of concept. It's a quite small change but would fully satisfy my use-cases. So if there is anything more I can do, to get this merged, please let me know (=

For anyone who wants to test this, here the commands to build and run docker-compose in a container:

# download and build
git clone [email protected]:acran/compose.git -b auto_up
cd compose
docker build -t docker-compose .

# create an example docker-compose.yml
cat > docker-compose.yml <<EOF
version: "2"
services:
  foo:
    image: busybox
    auto_up: false
    command: echo foo
  bar:
    image: busybox
    auto_up: false
    command: echo bar
    depends_on:
      - foo
  baz:
    image: busybox
    command: echo baz
EOF

# start all default services only, i.e. baz
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -ti -v $(pwd):$(pwd) -w $(pwd) docker-compose up

# start service foo only
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -ti -v $(pwd):$(pwd) -w $(pwd) docker-compose up foo

# start service bar, foo is pulled in as a dependeny
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -ti -v $(pwd):$(pwd) -w $(pwd) docker-compose up bar

# clean up all services, i.e. foo, bar and baz
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -ti -v $(pwd):$(pwd) -w $(pwd) docker-compose down

[acran]

@acran, I'd like to help you to update related docs and test cases. Is it ok ?

@dattran-vn01 of course, I would be happy :smiley:

[dnephin]

@acran in your example the only command that has different behaviour from the current Compose is the first one, and that behaviour can be accomplished using up baz.
You can also accomplish the same thing by using multiple Compose files.

Having some commands operate on a different list of default services seems very confusing, and has been one of the reasons I don't like this feature.

[acran]

@acran in your example the only command that has different behaviour from the current Compose is the first one, and that behaviour can be accomplished using up baz.
You can also accomplish the same thing by using multiple Compose files.

@dnephin yes, exactly, I know, and in this example it would be the easiest way.
But imagine this with 10 services and only one one-off service (like "init" or "backup"/"dump"): in that case you'd have to list all the services on the command line except that one single one-off service like docker-compose up service1 service2 ... service10 (see below). Especially you'd have to remember all the services and manually keep track of any changes in the docker-compose.yml.

I also know about the possibility of using multiple YAML files, see my first comment above, and this as well as explicitly giving all services on the command line does indeed cover most use cases stated here; but it feels more like a workaround to use.

Here another example docker-compose.yml to visualize the problem:

version: "2"
services:
  init:
    image: busybox
    auto_up: false
    command: echo "I do one-time initializing"
  service1:
    image: busybox
    command: echo "I'm service #1"
  service2:
    image: busybox
    command: echo "I'm service #2"
  service3:
    image: busybox
    command: echo "I'm service #3"
  service4:
    image: busybox
    command: echo "I'm service #4"
  service5:
    image: busybox
    command: echo "I'm service #5"
  service6:
    image: busybox
    command: echo "I'm service #6"
  service7:
    image: busybox
    command: echo "I'm service #7"
  service8:
    image: busybox
    command: echo "I'm service #8"
  service9:
    image: busybox
    command: echo "I'm service #9"
  service10:
    image: busybox
    command: echo "I'm service #10"

A simple docker-compose up _with_ the auto_up attribute starts here all services except service init. To achieve the same without it would require typing a much longer command with 10 services or splitting the YAML file and having to specify multiple files.

So the feature requested here is more about convenience and typing less on the cli and not about a completely new feature.

[ncoghlan]

Regarding the "use multiple config files" option (which is indeed the best currently available option), the usability problem with that is that it looks like this in practice:

$ docker-compose up one-shot-command
ERROR: No such service: one-shot-command
$ docker-compose up -f docker-compose.yml -f docker-compose-utils.yml one-shot-command
# Actually goes & does the requested thing

The discrepancy then infects _all_ of your project documentation and automation: you have to ensure _two_ compose files are available to anyone that wants to run utility commands, you have to ensure the full invocation is given for commands in the utility file, and if any command or service is ever demoted from "always running" to "only run when explicitly requested", you need to go find any invocation that doesn't supply the config file names and add them.

That is all _doable_, it's just annoying compared to being able to say in the compose file "Only run this when I explicitly ask you to, never run it by default".

[jakajancar]

Someone once told me: _"If several smart people cannot decide on what approach is the best, usually neither is pretty bad. The worst is not doing anything."_

This card seems a like a prime example of this :)

[hugochinchilla]

Given that docker now supports the concept of service natively for long running commands I think the best approach would be to add a new section called commands which allows for the same functionality than services except for restart policies, which should be disallowed.

[ootwch]

In my setup, I have services in the docker-compose that are required to run my infrastructure (nginx, database, webserver, message queues...). I also have define additional services that I only need for debugging reasons (for example a database web gui).

I would like the "debugging" services to NOT start automatically, but if I want to add them I can do so with a simple docker-compose up -d database-gui and it just gets added.

Also: Sometimes I change my mind and want one of these services to always start... => With option 2) I can just change that one flag

=> Vote for option two, because it is simple and it seems to satisfy everyone's requirements here. Every other solution feels like over-engineering to me... Adding complexity to the configuration based on rare or only imagined use cases, not on practical need.

[MadWombat]

Just here to add my +1. I am running on Django apps and when running them in docker it would be nice to have a container that would run shell or tests or for that matter migrate commands. But I do not want to run any of these every time I start the app. While using multiple config files would work, it would require typing out, scripting or aliasing mile long command lines. If I wanted to do that I wouldn't use compose at all, I would just shell script my containers (or python script them... maybe add a unifying YAML file to store container configurations... wait...) I don't want to do anything like docker-compose -f common.yml -f dev.yml -f local.yml -f commands.yml up migrate just to run database migrations on my container. The alternative is to use /bin/true as command and do something like docker-compose -f common.yml -f dev.yml -f local.yml up commands 'python3 manage.py migrate' which not ver elegant either. So having one time commands stored somewhere in the configuration would be very useful to me. Any of the options 2, 3 and 4 would work for me.

[dnephin]

I'd like to suggest you take a look at dobi, which is a tool I've been working on based partially on feedback from this thread.

compose is designed to start long-running services that form an "environment".
dobi is focused on sequential project build taks.

The things you're looking for (running a shell, running unit tests, running migration) are all project tasks, so they fit much better in the dobi model.

dobi and compose work quite well together, you can launch compose from dobi using a compose resource.

compose=dev-environment:
  files: [docker-compose.yml, local.yml]
  project: 'theprojectname'

which you can run using:

dobi dev-environment

There's an example of the compose integration and running db migrations here: https://github.com/dnephin/dobi/tree/master/examples/init-db-with-rails

There are examples of many of workflows (including running tests, starting an interactive shell, releasing, and building minimal images) here: http://dnephin.github.io/dobi/examples.html

[jan-zajic]

+1 for 2) . In our point of view It's needless to think and discuss uncommon usecases like one-off commands, but we just want to have real services that are created and ready for use, but can be started on demand when needed (and default don't consume resources).

[MadWombat]

"uncommon usecases"? do you run database migrations? do you run unit tests? do you run lint? webpack?

[jan-zajic]

Yes I run but that can be achived and controled inside of containers, in my opinion. But I do not want to detract your view or demands, let's not make this discussion flame. Our requirements on docker compose are not necessarily in conflict.

[Twipped]

I just came across this thread trying to find a way to create a docker-compose container for a script that wipes out the database in my mysql container and reloads the schema from a downloaded dump. It's a very lengthy and destructive process, so I don't want it running every time I bring up the services, but it also needs to run inside the composed set of containers using its own dockerfile.

Call them tasks, commands, whatever, but I'm kinda SOL without it.

[Salketer]

I've stumbled on this looking for something else, just wanted to chip in a use case... We found that our developers were the most capable of defining what data should be backed-up from their services. As the docker-compose.yml file is in their hands and named volumes are used, we decided to use it to define the back-up strategies... Here's a sumarized example of our old docker-compose.yml we were using.

version: '2'
services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    restart: always
    ports:
      - "5555:5555"
    volumes:
      - gitlab_config:/etc/gitlab
      - gitlab_logs:/var/log/gitlab
      - gitlab_data:/var/opt/gitlab
      - certificates:/etc/gitlab/ssl
      - registry_data:/var/opt/gitlab/gitlab-rails/shared/registry
  gitlab-runner:
    image: gitlab/gitlab-runner:latest
    restart: always
    volumes:
      - certificates:/etc/gitlab-runner/certs
      - /var/run/docker.sock:/var/run/docker.sock
volumes:
    gitlab_config:
      driver: local
    gitlab_logs:
      driver: local
    gitlab_data:
      driver: local
    certificates:
      driver: local
    registry_data:
      driver: local

To be able to back-up the volumes data, we decided to go with a busybox container to tar the data needed, but it has to be flexible and do only the volumes the developers want backed-up. Finally, we should be able to back-up every volume separately. To achieve this, we added 3 services to all our docker-compose.yml

  boot:
    image: busybox
    depends_on:
      - gitlab
      - gitlab-runner

  backup:
    image: busybox
    volumes:
      - gitlab_config:/data/gitlab_config
      - gitlab_data:/data/gitlab_data
      - registry_data:/data/gitlab_data
      - /backups/latest:/backup
    command: find . -type d -maxdepth 1 -mindepth 1 -exec tar cvf /backup/{}.tar {}  \;
    working_dir: /backup

  restore:
    image: busybox
    volumes:
      - gitlab_config:/data/gitlab_config
      - gitlab_data:/data/gitlab_data
      - registry_data:/data/gitlab_data
      - /backups/latest:/backup
    command: find . -type f -iname "*.tar" -maxdepth 1 -mindepth 1 -exec tar xvf {} \;
    working_dir: /backup

The backup and restore services just require to be configured using the volumes, the commands will do the rest. We are planning on adding a little bit of configuration to be able to choose which volumes to back-up or restore but for now, we do them all... Since we don't want those 2 services to start on every docker-compose up command, we needed to specify all the services we wanted, which can be tedious... So we added a dummy service called boot, all it does is depend on all the services that need to run and we just call this one when we docker-compose up.

This here is full of small hacks, but allows us to easily run docker-compose up backup to have our back up stored on the host's /backups/latest and from there we run our versionning/pruning logic.

I hope it helps someone who's trying to achieve something similar... But in the end, being able to just tell docker-compose to not boot those 2 services, we wouldn't need for a 3rd one and complicate our docker-compose up commands.

[schmich]

In development, we currently have 4 docker-compose files:

• docker-compose.yml: Defines the core services needed to fully operate our app, e.g. redis, MySQL, php-fpm, Laravel queue processor, nginx, Solr
• docker-compose-utils.yml: Defines utility services needed to run dev tasks, e.g. gulp, npm, artisan (Laravel), composer, redis-cli, usually run with the services above
• 2 more docker-compose files to define the environment and volumes for the services above

This setup works well since, outside of Docker, we have minimum dependencies for our dev environment. The drawback is the unwieldy command-lines needed to specify all the necessary .yml files to run simple utilities.

We develop on macOS, Windows, and Linux. In addition, many of our dev tasks require some kind of parameterization (e.g. creating a DB migration, running a composer/artisan/npm command with its own arguments), so I created schmich/runx as a zero-install cross-platform task runner to bundle up our common development tasks.

For example, this lets us say runx migrate:make create_some_table to effectively run docker-compose -f docker-compose.yml -f docker-compose-utils.yml -f docker-compose-dev.yml -f docker-compose-utils-dev.yml run --rm artisan migrate:make create_some_table, or runx npm outdated to view our outdated JS dependencies.

I haven't used dobi mentioned above, but runx feels similar in spirit, though less tailored to a Docker environment. Ultimately, I want to avoid dependencies everywhere except where absolutely needed: for the services themselves, and for the utility scripts needed in development.

[Iazel]

Any news? I think this feature would be really nice to have! I vote for the solution of a toplevel config like default where we can specify what service has to be run.
The boot dummy service is nice, however I prefer to leave the process in the foreground, therefore when I'm done with this work I can simply close the terminal to bring everything down. It's also nice to have the output attached by default and this doesn't happen with the dummy service.

[vipseixas]

Since noboby commented use cases using .env files, here is mine:

I have 4 compose files: docker-compose.yml, docker-compose.local.yml, docker-compose.prod.yml and docker-compose.ci.yml. For each environment I have a different .env:

CI .env:
COMPOSE_PROJECT_NAME=foo
COMPOSE_FILE=docker-compose.yml:docker-compose.ci.yml
WEB_PORT=8081
...

Test/dev/etc .env:
COMPOSE_PROJECT_NAME=foo
COMPOSE_FILE=docker-compose.yml:docker-compose.local.yml
WEB_PORT=80
...

And so on...

Users and/or scripts don't have to worry about which compose files to use since environments are already individually configured.

But I have a maintenance instance to run eventually, it cannot be configured inside main docker-compose.yml and one solution would be to create a docker-compose.worker.yml and execute a command line with 3 "-f" options that changes with each environment. That's too error prone to be an usable option.

The solution I found was to create a directory "worker", put the yml file there, link the upper .env file to this directory and create empty docker-compose.[ci|local|prod].yml files (otherwise I would have to change the value of COMPOSE_FILE variable).

This works fine, but it is far from optimal. Any other solution from the OP would sove this without any mess.

[acran]

I think the discussion here, the many example use cases as well as the fact that there are frequently duplicate issues to this opened makes quite clear that there _is_ a real demand by many user for a possibility to "Define services which are not started by default".

Therefor I'd really like to move the discussion here from _if_ this should be implemented more to _how_ this should be implemented. Could we maybe all agree on this?

Regarding on _how_ this could be achieved @bfirsh listed some options in the opening comment:

1) Recommend users to use a separate Compose file

This is the only currently available solution for this use case. But as said above it is in this case more of a workaround since it requires manually typing long command lines and remembering which Compose files to include for which command to run.
Of course, in a CI setup or automated deployment to production this is not an issue, since the command line has only to be defined once and never be typed out manually. So this is solely about convenience in non-automated environments, i.e. development, which is according to the docker-compose manual one of the main use cases of Compose itself:

Together, these features provide a convenient way for developers to get started on a project. Compose can reduce a multi-page “developer getting started guide” to a single machine readable Compose file and a few commands.

Having to type something like docker-compose -f docker-compose.yml -f docker-compose-utils.yml run init just to initialize a local setup does not sound _convenient_ to me but more like something to be found in _a multi-page “developer getting started guide”_.

And to consider option 1b) here:

Recommend users to use a wrapper script/tool to use separate Compose files

If I have to pull in additional tools and dependencies just for this, why use docker-compose in the first place? Since everything docker-compose does could - in theory - also be done with just a custom shell script and the native docker client directly.
I see the main advantage of docker-compose in it's standardized user interface (I clone a project, see a docker-compose.yml in there and know I just have to run docker-compose up to get started). Always having to know which compose file to use for what weakens this advantage drastically.

2) Add an option to services to make them not start by default
3) Add a top-level configuration option to define the default services

These two options do basically the same: define which services should be (not) started by default in one compose file, option 2) by blacklisting individual services, option 3) by globally whitelisting a set of services.
Considering that most of the time there are more default services than "one-off services" in a single compose file it's probably easier to blacklist one service than having to whitelist all the other ones and having to update the whitelist every time a default service gets added or removed. Also with option 2) merging multiple compose files is probably easier.

4) Add a concept of a thing like a service, but is just for one-off commands ("scripts", "tasks", etc...)

I think option 4) is just a specialization of option 2) or 3) since in most of the examples above the desired one-off containers are basically defined just like any other service and do nothing more or less like normal services - except that they're not started by default. (Or did I miss some significant difference to a "service").

So in my opinion option 2) is the best pick here, since it requires the least change but at the same time is flexible enough to satisfy most (or all?) use cases mentioned here and be very convenient to use at the same time.
Nevertheless I would be equally happy to see anything like option 3) or 4) merged into upstream. So after over a year of discussion in here, is there any timeline when or if this will happen??

[MadWombat]

I agree that option 2 is probably the least invasive. The only feature I see that option 4 might provide is an ability to run commands on existing containers (using docker exec mechanism). For example, if all I want is to run a database migration or unit tests, there might be no reason to create a whole separate container for it if I already have a container running my app. That said, containers are lightweight and I would be perfectly happy with option 2.

[mathroc]

@MadWombat I have the same feeling about option 4, that's why I proposed another feature for this use-case: #4096. there is overlap with the feature described here, but I believe it could be useful on its own or additionally to this one

[MadWombat]

@mathroc I mentioned option 4 using docker exec, since @acran asked if there was any significant difference between options 2 and 4. I cannot, however, come up with a real life use case where you would specifically want to run a command on an existing container rather than bringing up another similar container for the purpose. There might be some scenario with limited persistent connections to some service or some other limited resource that is already being used by existing containers, but I cannot think of one.

[acran]

The only feature I see that option 4 might provide is an ability to run commands on existing containers (using docker exec mechanism).

That's indeed a new use case which would justify (and require) introducing a new top-level concept different from services. But as long as the commands don't need to be run via docker exec within the already existing service container, the existing service concept should be sufficient.

I cannot, however, come up with a real life use case where you would specifically want to run a command on an existing container rather than bringing up another similar container for the purpose.

The closest case I can think about is something like changing a config file within the container and triggering a reload by the application; using @mathroc's example from #4096:

version: "2"
services:
  my_service:
    image: my/service
shortcuts:
  start-debugging:
    service: my_service
    command: echo "debug=true" > /etc/myapp/conf.d/debug.conf && mydaemon --reload-config
  stop-debugging:
    service: my_service
    command: echo "debug=false" > /etc/myapp/conf.d/debug.conf && mydaemon --reload-config

Although related to this issue, #4096 has it's own use cases and maybe option 4) could best be moved there?

[rawkode]

It's been over a year of discussion and we seem no further forward making a decision. @bfirsh should just pick a style and go with it :smile:

[Napas]

+1

[rysiekpl]

+1, that would be a very useful feature to have.

And another +1 for option 2, with something like auto-start: false or enabled: false.

My use-case is extending services. It is not possible to extend services that have depends_on set, so what I would like to do is to define a service template, and then extend several services from it. Obviously, the template should be built, but not run autmoagically.

[jonathanborges]

I created my own solution, get free to use.
It's still in Beta :)

https://github.com/jonathanborges/picles-docker

[charneykaye]

Lots of great options here. The need for the feature is real.

[g8d3]

Option 2 is clearly the more popular, occurrences:

1 -> 4
2 -> 52
3 -> 3
4 -> 10

[creynders]

Another 👍 for option 2

[briceburg]

We can handle this fairly easily in a "tools" service with an initial entrypoint or command that simply returns. Then we run this service with an alternative command to do what we want (e.g. tests or seed the data). E.g.

docker-compuse up -d
docker-compose run tools tests

Yes, the tools service will show as "stopped" in docker-compose ps. If this is unacceptable, then I the multi-file option from https://github.com/docker/compose/pull/2051 works. e.g.

docker-compose -f docker-compose.yml -f docker-compose-tools.yml up -d
docker-compose -f docker-compose-tools.yml logs -f tools

Although it looks like a lot of orchestration helpers are being written for this need and I'm guilty of working on one as well :) .

Personally I believe there's enough workarounds (such as the two above; getting creative with your entrypoints/containers and/or using the multifile approach with a shell alias/helper), and compose can be kept simple.

[acran]

@briceburg see the discussion above: the issue here is not about a missing functionality which couldn't be done any other way. There _are_ workarounds for almost everything people here want to achieve. But they're workarounds and dirty hacks instead of a normale use case of docker-compose.

It's about convenience, about not having to type out docker-compose -f docker-compose.yml -f docker-compose-tools.yml upby hand on the cli, it's about not having to remember whether it was docker-compose run tools tests or docker-compose run utils tests or even have to rely on a _multi-page “developer getting started guide”_ but instead having a unified and simple interface for the developer.

Again, CCing @bfirsh, @dnephin: could you please provide _any_ feedback to this issue? Is there any hope we could see something like this on the coming roadmap for docker-compose??

[vipseixas]

@briceburg for a simple environment maybe an workaround is acceptable, but sometimes it's not. Look at my setup:

rancher@rancher:~/servers/sgi/docker$ ls -l 
total 64
-rwxrwxr-x 1 rancher rancher  303 Dec  8 20:05 cleanup.sh
drwxrwxr-x 3 rancher rancher 4096 Dec 16 15:26 conf
drwxrwxrwx 4 rancher rancher 4096 Dec 15 20:03 data
-rw-rw-r-- 1 rancher rancher   94 Dec 14 19:40 docker-compose.amt.yml
-rw-rw-r-- 1 rancher rancher  295 Dec  8 20:05 docker-compose.auth.yml
-rw-rw-r-- 1 rancher rancher  332 Dec  8 20:05 docker-compose.ci.yml
-rw-rw-r-- 1 rancher rancher  112 Dec  8 20:05 docker-compose.dbext.yml
-rw-rw-r-- 1 rancher rancher  347 Dec 14 19:40 docker-compose.dbint.yml
-rw-rw-r-- 1 rancher rancher  688 Dec 15 16:31 docker-compose.full.yml
-rw-rw-r-- 1 rancher rancher   81 Dec  8 20:05 docker-compose.local.yml
-rw-rw-r-- 1 rancher rancher  288 Dec 15 16:31 docker-compose.yml
-rwxrwxr-x 1 rancher rancher  721 Dec 14 19:40 redeploy.sh
-rwxrwxr-x 1 rancher rancher  861 Dec  8 20:05 setup.sh
-rwxrwxr-x 1 rancher rancher   66 Dec  8 20:05 shutdown.sh
-rwxrwxr-x 1 rancher rancher  269 Dec 14 19:40 startup.sh
drwxrwxr-x 2 rancher rancher 4096 Dec 14 19:40 worker

I have 8 compose files (that I use in multiple combinations) besides the ones that are in the worker directory, where I configure the 'do not start by default' containers. The worker directory (with an .env file linked to the parent and some empty compose files for compliance with the parent's .env COMPOSE_FILE variable) is my workaround, but my life will be much easier when we have a solution for this issue.

[briceburg]

@vipseixas it's hard to gauge the sanity of this solution from a file listing -- although yes, I do agree having the ability to not auto-start a service is OK. I'm only suggesting that you use a default entrypoint/command that does nothing for those services you prefer _not_ to autostart. E.g. the container simply exits by default... and to execute those services with an alternative command via subsequent docker-compose run.

In most cases you'll want to ensure health-checks pass on dependent services/containers before executing tests -- for instance jenkins is UP and running (accepting connections on :8080 or whatever) before these tests execute. In the past I used dockerize for this.

If I were to propose a silver bullet, it would be to allow setting a compose profile via environmental variables (e.g. DOCKER_COMPOSE_PROFILE="tests" ), and to allow services to be dependent on a profile -- where all services default to the "default" profile or whatever. I would also ensure services that are dependent on another service are also _dependent_ on that services' HEALTHCHECK. So something like

app:
  image: myapp:v1

tests:
  image: myapp:tests
  dependencies:
    - profile: "tests"
    - service: "app"
      healthcheck: true

# run the things
docker-compose up -d

# test the things
DOCKER_COMPOSE_PROFILE="tests" docker-compose up -d
DOCKER_COMPOSE_PROFILE="tests" docker-compose logs -f

[charneykaye]

+1 for option 2.

Many here have proposed interesting new features, but what is the core of the issue here? I vote for keeping it as simple as possible.

As it stands today, we assume that all services in our docker-compose.yml will be brought up on docker-compose up. We're all working with our own custom bootstrap scripts, makefiles, build tools, etc., and fortunately both docker-compose and docker expose an extensive API for management of individual containers once the kitchen sink is 'up'.

My only need, the core need that brought me to this discussion, is to specify that one of those services I've defined is not something that I want 'up' by default, for whatever reason.

What's the simplest possible feature that "gets it done" for all of us? It's a tiny update to the docker-compose.yml specification ( @bfirsh original idea no. 2) wherein a service can specify autostart: false

[ghost]

I vote for: "JUST DO SOMETHING FOR US, THANKS." We vote, we now need at least one answer. 4 GOD!

[andersonDadario]

What's the simplest possible feature that "gets it done" for all of us? It's a tiny update to the docker-compose.yml specification ( @bfirsh original idea no. 2) wherein a service can specify autostart: false

That's way better than any other alternative :)

[drew-r]

It's so simple.
scale: 0
https://github.com/docker/compose/issues/1661

[ghost]

@drew-r what?

[gittycat]

@jonathanborges1542 It's docker-compose scale servname=2. There's no equivalent in the docker-compose.yml file.

@drew-r scale: 0 currently tells the docker daemon to send SIGTERM/SIGKILL to all of the containers for the service. This would kill any maintenance job even if it has not yet completed its job. That is the expected behaviour. Changing this would be a bad case of overloading what is a clear behaviour right now.

[ghost]

@gittycat I understand, but what we want here is to type "docker-compose up" and everything will be ready. Then in that case it would not work.

[gittycat]

I've been following this topic for over a year now (most posts are on the docker side).
I believe that a new top level "jobs" living alongside "services", "volumes" and "networks" is the best option. That's option 4 in the OP list above.
It's also a proposal over at the Docker repo.
https://github.com/docker/docker/issues/23880

[vipseixas]

@briceburg

it's hard to gauge the sanity of this solution from a file listing

The intention was just to show that I have many file combinations and using multiple "-f" options is too error prone to be done manually.

I'm only suggesting that you use a default entrypoint/command that does nothing for those services you prefer not to autostart.

Sometimes there's no escape from doom. But I prefer my solution: I use a different directory that will never be "composed-up" so there will not be useless stopped containers polluting my environment. That's why I have a worker directory with a ".env" linked to the parent ".env".

If I were to propose a silver bullet, it would be to allow setting a compose profile via environmental variables (e.g. DOCKER_COMPOSE_PROFILE="tests" )

THAT would be the best solution. But I don't think the developers are inclined to increase the complexity...

[Goram]

I want to build a base image, "extend" this in the same docker-compose.yml and start only the second image, that is not doable in one step at the moment. With option 2 it will be, but maybe the better and cleaner way is to have a "build stage" and a "run stage", where creation of base images is done. At the moment I have to run 2 commands and not just one.

[charneykaye]

+1 @drew-r #1661

[hadim]

Is is this issue going to be implemented at some point or is it just something that compose devs does not want to do ? It would be nice to know so the community can start looking for another solution.

By the way I vote for option 2.

Thank you

[drew-r]

+1 for option 2.
Would it respect explicit dependencies though... links etc ?

If I am specifying which service to bring up with docker-compose up [service] it is rare that I want to start any services other than the service in question and it's dependencies.

[msabramo]

Another vote for option 2. There's even a PR for it (https://github.com/docker/compose/pull/3047) that @acran was kind enough to create, so the effort from the compose devs should be fairly small, as they only need to review it (and it's pretty short and simple) and not write any code.

If the compose devs don't want to do this, then they should state it once and for all and then close the ticket and lock it to prevent comments.

[gittycat]

@msabramo compose is basically hamstrung by what is decided at the docker engine level.

The request is tracked over at the docker engine repo https://github.com/docker/docker/issues/23880
That's where the decision/work will come from.

I suggest that this issue be closed.

[mgriego]

@gittycat compose still can have the option of whether to tell engine to start a service or not. This is a very valid request still.

[jonathanborges]

I give up docker on production, there a lot of problems. Then i migrate to vagrant.

[drAlberT]

Another vote for 2)

[frnco]

Another vote for 2 and 4.

Just to point out a use-case, some services have circular dependencies, the need for generating config files, running test suites among many other possibilities. We can decouple it from Docker or use other tools but it adds unnecessary complexity to setting up the project.

Having a simple command for setup and/or for tests would make things a lot cleaner.

[sergioregueira]

Another vote for 2!

[ekkis]

I had read this thread but decided to create a new post (https://github.com/docker/compose/issues/4650), which got closed and I was referred here. the point I wanted to make is that whilst most of the discussion in this thread concerns flags to disable running containers, my use case is of a third state: creating containers.

I would like not only to prevent a container from being run but to be able to never create the container in the first place (in my use case I have a user-driven process that accomplishes this)

[frnco]

@ekkis I am curious about how that came to be. My usee-case involves referring to the main container image and configuring alternative "actions", could be done using something like bash but it makes more sense to use Docker.

Would you be so kind as to give more details on your issue?

[ekkis]

@frnco, not sure I understand what you'd like more detail on. there are cases where you want to build an image without needing the creation of containers. mine is one such case. I want a given image built along with a bunch of other images but this particular one is going to be used by a process I've written. I don't want to have two separate processes for building and having a flag for build-only could solve that problem

[frnco]

@ekkis I suggest you take a look at #963, seems related to what you're looking for.

[piotr-kwasnik]

+1 for option 2

[jcercurati]

+1 for option 2

[rhudson]

+1 for option 2

[y-fedorov]

+1 for option 2

[glyoko]

+1 for option 2

[adamcharnock]

+1 for option 2

[avkoval]

:+1: for option #2 with a question on any estimate on this feature? What is the best workaround? As far I can imagine need to list all my services like:

docker-compose up service1 service2 service3

... and omit those I don't want to start by default

[illegalnumbers]

Is there any way we in the open source community can help contribute this feature which seems heavily requested?

[nchicong]

+1 for option 2.
A way to specify 'stage' would be great. Say, the service will run only in production.

[damienlaguerre]

+1 for option 2.

[Alcereo]

Vote for option 2

[Tensho]

up command filtering will be perfect – docker-compose up service1 service3

[tinbka]

:+1: for options 2 and 4

[ianks]

At this point it seems apparent option 2 is what users want. What are the next steps for making this happen?

[coreyhickson]

Here to +1 for option 2, came across this issue hoping this functionality already existed.

[garceri]

+1 on Option 2

[LifeDJIK]

...and one more +1 for option 2

[elaijuh]

+1 for option 2

[synergiator]

+1 for Option 2 :-)

[podorozhny]

+1 for options 2 and 4

[arekborek]

Hi All,
+1 option 2

IMHO, option will be very useful on local environments, when you have application based on micro-services or with hexagonal architecture; as a developer I can run only necessary elements for my nearest work;

[AyushyaChitransh]

Option 2 is the one which most users want. Is the decision regarding this still to be made or development on this has started and being tracked somewhere else?

[MohammedNoureldin]

I think when I finish the university in a couple of months and have more time, it will be much faster to implement this feature by myself (although I haven't learned Python, in which I think compose is written), but I saw that huge amount of votes and requests, for two years, without any step forward, and self implementing seems to be the solution for the next 5 years (till that feature gets implemented officially).

of course +1000 for option 2

[LinBoLen]

+1 option 2

[com-quoininc-gmanning]

Would it make any difference if I voted for option 2? ' seems unlikely that it would.

[shin-]

The comments here have become acerbic as of late, and no new feedback has really surfaced, which is making me consider locking this thread. Please remain civil when interacting here.

This is the current, official position of the maintainers team on this matter. While it's still under consideration whether we want to add this feature, or one like it, as a convenience option, it is not a priority for the reasons highlighted, and because other tools exist that are more suited towards it.

[jimzucker]

@shin- Before we lock this can I as that we get clarity if the community is will to development would the core team merge it if it meets the standards of the team? In the threads there is an implementation and community member willing to complete it, they were just looking for some confirmation that it would not be a waste of their investment. If we can get some feedback I will track down those threads to move to see if we can move this to closure.

[shin-]

@jimzucker No - because the problem here is not the implementation ; a basic implementation of this would take half a day for your average developer. What we're concerned about here is a fundamental design issue.

1. What is a service? By design, a service is a clearly defined, long-running process that interacts with other services to form an application. That is the basic assumption that Compose operates under and it guides our design for most, if not all features we implement.
2. By that definition, a "service that is not started by default" is not really a service [1] - it's more of a pre-defined task, running in a container and acting on the application, rather than being part of it.
3. Hence it is wrong (fundamentally by design) to call it a service. At that point we need to consider creating a new type of definition in your Compose file (I'll call it "task" for brevity, the exact terminology does not matter here).
4. Since tasks are inherently short running processes, they use a different set of options compared to services. For example, we can intuit that tasks do not need to expose ports [2]. On the other hand, an option like CID file which is mostly pointless for services could be really useful for tasks.
5. Now, in most projects, tasks tend to have dependency chains (that's why in traditional, non-container world, we have Makefiles) - so we need to have at least a basic implementation of dependency chains, with parallelism, and being able to tell if that one JAR build task that always takes 10 minutes really needs to run again.
6. Then, some services actually depend on tasks having run before being able to start, adding another level of complexity to how we calculate and handle dependencies.
7. And some tasks need some services to be up in order to run (e.g. database migration).

At this point, this has become an entirely new piece of code that probably matches or surpasses cmake in terms of scope and complexity. This is why we've resisted implementing this for so long - partly because this is not at all what Compose is designed to do, but also because we [3] simply don't have the time or resources to implement or maintain what this feature would need to become to be truly useful to people.

Hopefully this clarifies things a bit more. I know people have been upset and disappointed with the response (or lack thereof) on this thread, and I'm not sure this will alleviate that in any way, but there it is.

[1] Some people do mention wanting to have "optional services", but that's not the most common use case mentioned in this thread. Most people want the "task" feature, as far as I can tell.
[2] I'm sure someone will contradict me with an example of a task that does need to expose ports, which will only go to show that the exercise of deciding which options are applicable to each set is not trivial.
[3] And at this point I mean "I", as I'm the only maintainer actively working on docker-compose at this time.

[jimzucker]

@shin- I see the point. What I am pursuing is "optional services" in your footnotes and the use cases for that, not the idea of tasks, can we somehow agree on how we can progress that point as that was the part of the thread I am trying to champion and my tickets clearly on that path where closed in favor of this thread. If we can separate that from the points you have clearly defined as 'tasks' we can make some headway. When people are doing a +1 for option 2 in my read it is to the optional service. Happy to take this offline to discuss and we can then come back to the forum, I am at [email protected]

[ekkis]

@shin- your point #2 is wrong. a service is a service regardless of whether it's running. there are countless use-cases where dependent services will modify behaviour when the dependency is not available, so your application layer may not, must not, require that all "services" be available at all times

if you think of it in the most traditional terms, consider that a web server is a "service" but it can be shut down or configured not to start by default on a given host but no one would consider it anything other than a service

[sergioregueira]

I think a lot of people that are requesting this feature do not want to run tasks but optional services. For example, if you have a set of microservices running as Docker containers and you deploy all of them as a whole service, maybe some day your requirements change and you do not need to run all the containers in all your servers because for that case you do not need a specific functionality that is included in a non mandatory container.

Additionally, i think this is a nice feature for development environment; if you are fixing a bug in a specific container you do not need to start all the containers for it.

[blop]

Exactly, with a simple boolean field we could easily enable/disable services from the .env file and avoid editing the yaml file.

[shin-]

@ekkis I think you're confusing fault-tolerance and dispensability somewhat. Also, see my foot-note on the subject, where I acknowledge some use-cases do exist.

Everyone: please keep in mind that if some services are optional for your application, it is already possible to handle this with current features. The recommended way to do so is to have a separate Compose file that can be included or excluded using the -f flag. They are not a very compelling argument for adding this option.

[jimzucker]

@shin- I appreciate you taking the time to have this dialog with us. I agree there are several ways to do it, as you suggest multiple yml files, you can also pass a list of services but these are not convenient and lead to user error creating operational risk. Especially in the case of a commercial product we want to distribute one compose file and by default get the desired common behavior and give instructions for advanced options, one file make all of this much neater for both dev, CI/CD and distribution. The addition of ENV variables with defaults was a huge help to us and has eliminated the need to have many yml files, this will be one more step towards managing a single file describing the deployment.

Is there any form that core team would be willing to accept a contribution on this LMK.

[ekkis]

@shin, I made no reference to either fault-tolerance or dispensability. I merely made a point about the definition of a "service", namely that whether the service runs or not is not inherent in the definition

[acran]

@shin- I strongly agree with @ekkis: not being started _by default_ does not violate the definition of being a service, therefore your point 2. is an assumption which is not _necessarily_ true and so are the consecutive points since they're based on the assumption of 2.

For example: when I develop an application using a mysql DB I usually define an additional _service_ phpmyadmin. This is a true service: long running, exposed ports, dependent on other services.
But I don't want do start this service _by default_ but only _on demand_/in development.
Currently I realize this by putting this service into an additional docker-compose.yml and starting it by providing both files to docker-compose - just as you suggested above.

So, yes, there already are possible ways to achieve this; be it by splitting the services into multiple files or simply using a tool other than docker-compose. But as I wrote already above: this issue is not about an entirely new feature/functionality or a new concept (i.e. _tasks_) but about convenience, especially in development environments.
Having to type out docker-compose -f docker-compose.yml -f docker-compose-tools.yml up is anything but convenient and so docker-compose does not deliver on its own promises:

Together, these features provide a convenient way for developers to get started on a project. Compose can reduce a multi-page “developer getting started guide” to a single machine readable Compose file and a few commands.

Which services to start or which yml files to use in which environment is something typically to be found in _multi-page “developer getting started guide[s]”_ - if to be found at all...

So in my opinion and as @jimzucker wrote most +1s here are really for "optional services" rather than a new concept of tasks. At least all of them are better off with a simple boolean to flag a _service_ to be started by default or not.

As my PoC shows the required code change - for "optional services" - is quite manageable and provides for most people here a huge benefit in convenience. Option 2 - read as "_possibility to mark a service optional_" - is a reasonable compromise to solve this issue.

@shin- wouldn't you agree on that giving users the possibility to mark services as optional - but them still being services - would greatly help them?
When reduced to "marking _services_ as optional/not started by default" would this issue/a pull request for this be accepted and merged?

[ekkis]

@shin, to add to @acran's point above, I may want to have docker-compose only build certain services because those services will be run by other services using names. consider that every time I do "up" I get a bunch of "services" named project_app_1, project_otherapp_1, etc. these make no sense as they need to be run using information in a database (I have a master service that kicks them off). so I always have to go kill them. so for me a NO_RUN or BUILD_ONLY flag makes total sense. this would be convenient

[shin-]

@acran

wouldn't you agree on that giving users the possibility to mark services as optional - but them still being services - would greatly help them?

I disagree on how much it would help - I think "greatly" is overstated. In your example, just having a docker-compose.phpmyadmin.yml that can optionally be included is no more burdensome or complex (in terms of instructions "developer guides") than setting an environment variable / modifying a service definition inside the main file.
I'm also considering the tradeoff of giving developers "footguns" when it's not strictly necessary to do so.

When reduced to "marking services as optional/not started by default" would this issue/a pull request for this be accepted and merged?

See above – as soon as this feature becomes available, people will start using it for things other than its intended purpose, and expect us to implement a new featureset to support it. Also cc @jimzucker ; I can't currently think of an implementation that would prevent this scenario from unfolding. If I could, this issue would have been closed a long time ago.

@ekkis If you never want to start those services yourself, why exactly is it an issue to have them isolated in a separate file? I'm really confused as to why this is such a hang-up here.

[ekkis]

@shin if a docker-compose.yml can be used to build a service, it is a most useful tool in centralising my build process. it actually makes a great deal of sense that I may want to (conditionally) build a service before running it, and I applaud that functionality

so the docker-compose file is also my makefile. but just because I build something doesn't mean I want to run it. it makes no sense for docker-compose to assume that I want to run a service, invent a name for it, and run it without proper parameters/environment variables/etc.

frankly I'm not sure I understand what the objection to this simple patch is. it would be greatly useful to me

[shin-]

I clearly pointed out in my earlier post why the team's official position was that you should not use your Compose file as a Makefile. If you believe we're wrong on this and still want to do so, that is your prerogative, but it is a bit preposterous to insist that we add features to support it.

[acran]

@shin-

In your example, just having a docker-compose.phpmyadmin.yml that can optionally be included is no more burdensome or complex (in terms of instructions "developer guides") than setting an environment variable / modifying a service definition inside the main file.

Actually I didn't think about involving environment variables here yet. My desired workflow "optional services" would finally allow is as follows:

• I copy/clone a new project
• I see there is a (single) docker-compose.yml file in it, therefore I know, all I have to do is run docker-compose up and the application is built and run
• Now I want to start one of the optional services for development purposes (e.g. additional debugging, phpMyAdmin). All I have to do is docker-compose up phpmyadmin.

For me this is _much_ more convenient than using multiple docker-compose.yml files, and as I understand for most other people here too. The benefit of having "optional services" here is

• for the developer not having to duplicate anything between multiple files or having to carefully consider which definitions to put in which file, but only having to set one boolean flag (auto_up: false) on a service
• for the user not having to know/remember which files to include for which services: do I need to run
  
  
  
  • docker-compose -f docker-compose.phpmyadmin.yml up
  
  
  • docker-compose -f docker-compose.yml -f docker-compose.phpmyadmin.yml up or even
  
  
  • docker-compose -f docker-compose.phpmyadmin.yml -f docker-compose.yml up?
  
  
• instead the only thing the user needs to _know_ is that there is a service called phpmyadmin and the user knows immediately how to start it.

See above – as soon as this feature becomes available, people will start using it for things other than its intended purpose, and expect us to implement a new featureset to support it

Don't get me wrong, I not only understand your concern here but agree with it: there will always be users who bend this - or any other (!) - feature to fit their desired workflows even though the features were never meant to be used like that.
BUT for any request on "optional services" missing features of a "task" you can _then_ easily argue that services are not tasks and tasks need to be introduced as a new concept first and are therefore not supported (yet). I.e. anything more than just marking a service as optional is a step towards the concept of tasks (e.g. option 4) and there is a clear cut between.

With the feature of marking a _service_ as optional we're still on the right side of that cut. "The feature per se is not too complex to implement but it _might_ tempt users to use it wrongly" currently being the only real (although valid!) argument standing against this feature is quite unsatisfactory as a reason not to implement it - at least for the users who could get much of convenience out of it while using it the correct way™.

I am up to rebasing my pull request #3047 to current master and do any required polishing for this to get accepted, please just let me know ;)

[rysiekpl]

My use-case is that we don't push all our images to docker hub or quay, nor do we have our own registry, and being able to mark a service as not-to-be-run-by-default means we can have a base image defined in docker-compose.yml that doesn't get started, and then other containers using that or building upon it.

I would be inclined to use a docker-compose fork that implements this feature just for this singular feature.

[creynders]

@shin- Though I think we all understand your point and sympathise with it, I don't think "it might be misused" is a good argument for something with - clearly - many viable use cases. Of course it _will_ be misused. That's exactly how software and new ideas grow; out of using technology not entirely as it should be. The web wouldn't be what it is today if we hadn't been misusing all provided technology for the past 25 years.

Anyway.
According to you - simplified - the difference between a service and a task is the life time. Makes sense, and I understand you view them as two different concepts and feel it's a potential can of worms. But this distinction is nowhere clarified or expounded upon in the documentation. Nor is it anywhere pointed out that docker is not meant to be used for spinning up temporal processes.
What I mean to say is that if the docker team has a strong stance about this, it should be made clear from the get-go.
Also. Task or service, there's clearly an overlap. And marking a service as optional is definitely not something that belongs uniquely to the task-side.
From a practical point of view I definitely agree with @acran - having a single docker-compose file is far better DX, simplifies a lot and is less error prone than having multiple docker-compose files.

[creynders]

Hehe, I remember the fierce discussions we had when we were lobbying for allowing "browsers" to display images, with the main argument of opponents being that it's called hyper text transfer protocol.
Sorry for the OT reminiscings of an old fart.

[ekkis]

@shin- if the docker-compose should not be used as a makefile, then it should not provide makefile functionality. if it allows me to build my projects (bravo!) then to take the position that it is not a makefile is ludicrous

[ekkis]

@shin- look, you guys don't have to implement any feature you don't want to, no matter how unreasonable your reasons, or how useful the feature. it's just that it makes the tool less useful to those of us that use it. we are the users, we know the use-cases -- and tools are only as useful, valuable as they cater to use cases

this to me is only a minor irritation, I have to manually kill a bunch of stuff that gets spawned for no good reason _every time I up my services_. to others it may be an impediment, the irritation will vary in severity

I understand that the official position is we should use dobi, even if it's overkill for what we need. but that official position does not explain the basis for your reluctance to accept a useful PR. we can still use dobi when the demand on docker-compose is too great or the overlap sufficient to migrate, but this is a very small change that would make all of our lives much better

and incidentally, I'm not being "acerbic" but locking the thread to prevent further discussion would simply be undemocratic and authoritarian. open source is built on the principle of openness and I would be sad to see this project managed using censorship and dictatorial tactics

[shin-]

@rysiekpl This is a problem we're looking at solving in a different way, see https://github.com/docker/cli/pull/452 (once syntax is deliberated, it'll make its way into Compose and v2 as well)

@creynders I think that's a bit of a mischaracterization - nobody is saying that optional services shouldn't exist (and in fact, as I pointed out, there are ways to declare them by using separate files). This is more akin to debating whether not explicitly closing an <img> tag is valid and whether our specific browser should support it or not.

Also, to your point:

But this distinction is nowhere clarified or expounded upon in the documentation. Nor is it anywhere pointed out that docker is not meant to be used for spinning up temporal processes.

This is actually the first paragraph of the https://docs.docker.com/compose/overview/ page (emphasis mine):

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a Compose file to configure your application’s services. Then, using a single command, you create and start all the services from your configuration.

@ekkis I'm really glad that Compose is in fact helpful for your projects! I hope I don't come off as insincere in all this, it's absolutely okay if that's the way you want to use the software. What I am saying, however, is that taking it further in that direction would probably make it worse in the long term. We all know what happens to applications that try to do too many things (although I guess Emacs pulled it off :stuck_out_tongue: )

I don't think I was being "authoritarian", but maybe I didn't express myself properly. My intention was to encourage people to remain civil and positive in their interactions. The thread is still unlocked and will continue to be as long as people refrain from being mean or impolite.

[ekkis]

you create and start all the services from your configuration.

and I guess our contention is that should have read "create and/or start", though to be fair to the language one needn't indicate the OR because it's implicit in the AND. in other words, it is true that you can both create and start a service but there's no implication that it must do both

We all know what happens to applications that try to do too many things

yes, I can appreciate that you're managing the borders of the domain

I don't think I was being "authoritarian"

I love the open source movement. we can have our disagreements but we're still in this together. the wars within the bitcoin community were appalling and I guess I'm still reeling from that, so thank you for staying on the side of civility

[ekkis]

as a parting thought I would ask that you re-review https://github.com/docker/compose/issues/4650 which is not the same as the request on this thread (as you indicated when closing it) but accommodates my use case, namely that the flag being discussed here isn't binary but trinary

[riedel]

I really like the idea of defining multiple files. However including all files with multi -fincluding the original seems to be a total overkill.

There is actually a more practical way, that works for me (see PS below): put all the service you don't want to run on up in docker-compose.override.yml. And use docker compose -f docker-compose.yml up for normal startup (note that this a one-time command). On all consecutive calls the override will be sourced...

Overall this is all very unintuitive and a bit unsatisfying. Adding an additional:

• an option for disabling services makes sense especially for more complex overrides

• The -foption is minimal but not very intuitive, another -F <name> option that just adds additional overrides would be very welcoming (i.e. docker-compose..yml. An alternative would be using as a directory in which all .yml files are sourced: together with symlinks this would be similarly powerfull as /etc/apache/sites.available -> /etc/apache/sites.enabled

On the other hand: anyone could provide a simpe wrapper script (function) for their shell to emulate this behaviour...

PS: my use case is that I define a debugger as a service. Obviously this service is not needed all the time but needs the links (also for security reasons) to work correctly.

[campadrenalin]

Another use case for which Option 2 is great:

Let's say you have a team of people working on the same project. Some of them have access to AWS managed services, but some of them need to run the corresponding resources as local containers. So depending on my workflow, I'll either _always_ or _never_ need Redis and MySQL containers in my compose constellation.

I'm going to flat-out state that managing multiple compose files is stupid for this use case. The amount of overlap is comical. Maintaining multiple files is a footgun - they _will_ get out of sync and cause confusion. I'm frankly a bit shocked that anyone took that option seriously for _any_ use case.

Being able to quickly enable and disable services, without commenting out large blocks of configuration (awkward in YAML, very messy if accidentally committed), is great on its own. But if compose someday gets the ability to interpolate variables into the config, this feature automatically becomes exponentially more powerful. Until then, it's _merely_ damned useful.

[rysiekpl]

Completely agree with @campadrenalin.

This whole discussion, as I understand it, is basically a large number of people providing a large number of very good reasons to implement Option 2, and the developers refusing because somebody hypothetically could potentially misinterpret the existence of this feature and request some other unrelated features.

[piit79]

OT:

But if compose someday gets the ability to interpolate variables into the config, this feature automatically becomes exponentially more powerful.

Hasn't this feature been available for a long time? See https://docs.docker.com/compose/compose-file/#variable-substitution. I'm using it to allow customization of our Compose dev environment by using the .env file - e.g. source code location, port mapping etc. Since compose file version 2.1 even the shell-style variable default (${VARIABLE:-default} / ${VARIABLE-default) is supported.

[acran]

This whole discussion, as I understand it, is basically a large number of people providing a large number of very good reasons to implement Option 2, and the developers refusing because somebody hypothetically could potentially misinterpret the existence of this feature and request some other unrelated features.

I find this to be - maybe a bit harsh in tone, but valid in essence - a very good summary by @rysiekpl.
So, @shin-, did we come any closer to a decision regarding this issue? Whether to just close it or to implement/accept a PR for option 2?

I totally see and understand your concerns, but also see the advantages outweigh the risk of someone potentially abusing this, since it is a rather small change but offers a lot of people much more possibilities for much more _convenient_ workflows.

[blop]

As docker-compose seems to be dropped in long term in favor of docker stack anyway, I suggest everyone moves to the new v3 format and docker stack tool as soon as possible.

Using a stack on a swarm (even on 1 node), you can specify this:

        deploy:
            mode: replicated
            replicas: 0

Having zero replicas essentially disables the service from running.

You'll just need some other tool to generate that yml if you want to apply environment variable substitutions, like envsubst.

You can then use a placeholder in your yml :

        deploy:
            mode: replicated
            replicas: ${R}

And process it this way :

export R=0
docker stack deploy stack1 --compose-file=<(envsubst <docker-compose.yml) 

[rysiekpl]

So, if docker-compose is going to be deprecated in the long run, what exactly is the problem with merging the pull request that is already out there? I mean, if docker-compose is going the way of the dodo, how is "somebody will request weird features" a real problem?

[stefanotorresi]

As docker-compose seems to be dropped in long term in favor of docker stack

So, if docker-compose is going to be deprecated in the long run

wait, wat? docker-compose is central to a lot of people for their development workflow, which hardly needs a swarm manager. Could you point me at some official statement by Docker on this regard?

[MadWombat]

I might be reading this wrong, but it sounds like docker swarm and docker stack are tightly coupled with docker cloud services. Are these things configurable on a standalone, offline system?

[stefanotorresi]

Are these things configurable on a standalone, offline system

yes, you can use them offline.

[SolomonShorser-OICR]

To me, it looks like stacks on docker-swarm will eventually be a superset of the features of docker-compose. I think the descriptor files are already very similar. Hopefully, this transition will not be too difficult. The biggest annoyance that I can see is managing the swarm separately from the stack(s). More overhead work for me since I have no interest in running composed applications across more than one node.

Or maybe I'm misunderstanding everything...?

[tberne]

+1 on option 2!
And I think the drop of docker compose in favor to swarm stack will be a big drawback to the community. Docker compose is so simple and easy to use!!

[ekkis]

@tberne I agree with that. docker-compose is simple and simple is a good thing. we should vote to keep it

[OnkelTem]

Heh, what an extensive discussion! Don't pay much attention to this comment, just my two cents.

At first I agreed with the majority to just add a sort of enabled flag.
Then I saw a proposition to branch definitions by groups and considered it as a good idea.
Then I read about Dobi, tried it, ensured that it can't be used in my project right now and went on reading.
Then I read arguments of @shin- and deemed them good enough, despite I also agreed with some counter-arguments.

My current solution is to create some docker-compose yamls and write some bash wrapping scripts to not run docker-compose by hand.

E.g. this is how I run containers which run a website (./docker/scripts/up script):

#!/usr/bin/env bash

def_opts="-d"

cd "$(dirname "$0")"/../

# A separate bash file with functions like: get_args, add_hosts...
. ./functions.sh

docker-compose -p ${PROJECT} up ${def_opts} $(get_args "options services" "$@") && \

# Adding host to /etc/hosts
add_hosts "$HOSTS" web && \
add_hosts "$HOSTS_MAIL" mail

[CpuID]

Big +1 to either Option 2 or 3.

[kitingChris]

+1 for Option 2 or 3.

Also the proposed "group" concept sounds interesting!

[gregunderwood]

+1 for option 2

[luckydonald]

_There are so many +1 on option x, that I can't even find the original statement. Please use the reaction buttons under posts instead._

To add a new idea:

Can't we just set a default scale to 0?

web:
  image: busybox:latest
  command: echo 'scaled'
  scale: 0

[joeyhub]

You should just do option 2 (but maybe nostart instead or something) because people have been waiting years now, then wrapping docker-compose with their own configuration managers because the tool is too limited, which defeats the point as it's doing what docker compose should be doing. In two years someone could create their own more powerful docker-compose as well. That's a lot of time this issue has been open for.

In the long run there should be some consideration of true options grouping. Some are relevant only to build, some only to run and some are mutual (you really want a layer of nesting at least for run or just separate out and have a build section, problem solved). Currently though a lot of things are problematic in the configuration, I'm all for just remake it all in version 5 but getting it right that time (primarily by representing the real on a technical level domain rather than trying to make it map a usage based domain or overly simple, instead create a config layer for those goals on top of the right one, you can't do it in reverse). I think because of that state of affairs the quick fix is justified.

[pjxiao]

+1 for option 2 or 4

If there aren't any plans to add this feature currently, I want the way to run such services (or tasks) in a composition is written in docker-compose's docs.

[paulodiovani]

This is an old thread, and I agree that docker-compose shall not care about _services that won't run from start_.

But I also commonly add a test service, so here is what I did:

1. Override the entrypoint of the service.

2. In the new entrypoint, check if there is a tty available (docker-compose run uses --tty by default while up doesn't).

   # Check if there is a tty
   if [ ! -t 1 ] ; then
     echo "No tty available."
     exit 0
   fi
   

   This will always make the service exit when run without a tty (with docker-compose up, for example) while still runs as expected with docker-compose run

[z3ntu]

@paulodiovani That looks more like a hack than a solution.

[bijayrungta]

+1 for a feature request.
hacks like having commands that do nothing means even if the container doesn't run, it has to still be built, initialized etc..

[neumannjan]

I solved this in my project by providing a simple bash script that calls the appropriate docker-compose command based on provided arguments.

Now if I do ./docker-compose.sh -e testing up, the result is the same as if I did docker-compose -f docker-compose.yml -f docker-compose.testing.yml up.
If I just do ./docker-compose.sh up, I get, as you'd expect, regular docker-compose up.

Therefore, my -e ${environment} option (a short for --env) is basically an alias to -f docker-compose.yml -f docker.compose.${environment}.yml. This results in docker-compose.yml being the base/default compose file and files like docker-compose.testing.yml being its extensions.

This somewhat solves @acran's problem of the user not knowing which files to include (and in which order) to the docker-compose command by enforcing sane defaults. It is not a very robust solution as some may use more complex compose file combinations, but I think docker-compose should be able to do something similar. Most docker-compose files are likely to start with docker-compose. and end with .yml anyway, so why do I always have to type out such a long command (docker-compose -f docker-compose.yml -f docker-compose.testing.yml up) if I just need a single additional service for my testing environment?

[oppianmatt]

No need for envsubst, docker-compose config does the environment variables and then docker stack takes it from stdin:

docker-compose config | docker stack deploy --prune --compose-file -

Let's you use the docker-compose .env substitutions as well.

[frnco]

@kogli I think the problem here is to pick an option. All have Pros and Cons. As I said before:

Every solution will be better for some use cases, and worse (Or even useless) for others. If the something is implemented, the mantainers it forever, so I believe they're wise in taking their time with it.

@shin- I guess you're the one to ask: Does it even make sense to consider this? Considering the current position of the team, how hard it is to design it, and the many tools out there, wouldn't it be easier to simply tell people this won't happen? Most people if not everyone who commented here found a way to deal with it.

Closing this seems pretty good, plus you can focus on the stuff that matters, and no one wonders if they should make a pull request and how to meet all those needs at the same time. They can just go build something else, and maybe even publish it to help others.

Just for the record: My solution was not perfect, but way better than the 2 days it took me to install the dependencies and fix a shitload of stuff and make the project run (and it already used Docker). Just 3 commands, with Docker and Docker Compose as the only dependencies. Yes there was more code than I wanted. Yes there's a shitty order to run commands. And yes there's a tiny bit more to keep in mind. A Wiki page covered those. Preety good, if I do say so myself, which means Docker Compose did it's job, I just also needed somethig else.

Point is: If it's out of the scope, it's a waste of time to keep discussing it. Many people just stop here to ask about this and give their opinion, before trying to decide what to do. They will still find or build something else if this is closed, just faster. And you get one less thread to babysit.

Edit: Just thought of a good way to put it: If there's no clear criteria to accept PR and there is no one in the core team either involved with it or planning to take a look at it in the foreseeable future, it's not an "issue", just an idea.

I see no reason to clutter the issue list, I believe many ideas here are good, but I think they belong somewhere else, for good or for worse. It's almost 3 years, and on the off chance that one day the situation changes, you can always reopen this, or open a new issue about it.

[rubycut]

This feature takes like one day to implement because it's really simple. However, after 2 years of discussions, still nothing. Sad.

[piit79]

@rubycut Yeah, that's what happens when so called "higher principles" take over from usability and user-friendliness. I completely understand the devs don't want to clog the product with illogical features. However, this particular highly requested feature actually seem to make sense. It's sad when the developers think they know better than their users. (Again, they usually do. But not always. Not this time.)

[9nix00]

@rubycut Talk is cheap, show me the code. If you can,send your PR, if you can't, just report it and wait .

[rubycut]

@wangwenpei Are you kidding me? First of all, you are not maintainer of this project. If maintainers would accept this idea, and put "help wanted" tag, I am sure somebody could contribute pull request. But they said that they are not interested. So, why bother will pull request if it will not be accepted.

[alecmev]

Just FYI, "the code" is already there: #3047

[acran]

@wangwenpei well, #3047, there's the pull request...
waiting to get at least some feedback on it for over two years now. It was closed recently

for the reasons highlighted in https://github.com/docker/compose/issues/1896#issuecomment-322285976

which quite angers me to be honest because the arguments given in https://github.com/docker/compose/issues/1896#issuecomment-322285976 do not relate in any way with the pull request:

1. What is a service? By design, a service is a clearly defined, long-running process that interacts with other services to form an application. That is the basic assumption that Compose operates under and it guides our design for most, if not all features we implement.
2. By that definition, a "service that is not started by default" is not really a service [1] - it's more of a pre-defined task, running in a container and acting on the application, rather than being part of it.

I don't see the assertion By that definition, a "service that is not started by default" is not really a service to be true. A service that is not started by default is just a service that is not started by default, nothing more, nothing less at this point. Being started by default or not doesn't say anything about the service being long-running or not, being part of the application or not, it only says that is not to be started by default.

And that's also exactly what the change in the pull request does: add the ability to mark a service to be started by default or not.
There is no need to introduce a new concept such as tasks to do this. In fact all the other arguments in https://github.com/docker/compose/issues/1896#issuecomment-322285976 follow from the false premise of 2. and are thus not applicable to the pull request which does not try to do anything else.

Just for completeness:

[1] Some people do mention wanting to have "optional services", but that's not the most common use case mentioned in this thread. Most people want the "task" feature, as far as I can tell.

Yes, there were examples of one-off tasks, but there also were examples for _real_ services according to your definition, i.e. legitimate use cases for this feature.
Being able to use this to _also_ run one-off tasks I can not really accept as a valid argument against this. It is already possible to define a _service_ which is not long-running, not exposing any ports and is not an essential part of the application in a totally valid docker-compose.yml.

In fact this is also one of the suggested workarounds! Just split the _tasks_ into a separate docker-compose.yml.
The other workarounds mentioned are to use your own home-made (platform-dependent) scripts, which totally contradicts the self-stated purpose of docker-compose, see my comment above, or to simply switch to another tool, i.e. throw away your perfectly well working toolchain just to get this one tiny feature in convenience which could easily be implemented in your current toolchain (docker-compose).

The only real argument brought up against the most basic implementation of option 2 - i.e. having _some way_ to tell docker-compose to not start a particular service by default, and nothing more, no new concepts or anything - is that this could be_somehow_ be misused by _some_ users.
But this argument is quite weak compared to the demand for it and the legitimate use cases brought up here...

@shin-

but also because we [3] simply don't have the time or resources to implement or maintain what this feature would need to become to be truly useful to people.
[3] And at this point I mean "I", as I'm the only maintainer actively working on docker-compose at this time.

On the one hand highlighting that there is only one active maintainer while on the other hand just ignoring pull requests and therefore wasting any opportunity to maybe get other developers involed in the project isn't something I could comprehend.
The code changes provided in #3047 already _do_ implement a feature which would be _truly useful to [many] people_ - maybe not _all_ the people with _all_ of _their_ use cases, but most of the people here.

All the additional requirements - having to introduce completely new concepts, new dependency management, up to a cmake-like complexity - simply follow from a false premise!

[Songworks]

I came upon this issue just a couple days ago because I was looking for a feature like this.

Since it's unfortunately not implemented, and from what I could gather, the officially supported alternative is to use multiple yaml files.

Well, I find the idea of using multiple yaml files for services not run by default a poor alternative in comparison to using one yaml file for all services containing one additional setting for each service not run by default.

So, here's my +1.