Compose: Data Only Containers/No Run Containers

[kojiromike]

I propose that Compose be able to support a way to create volume containers that never run. Right now, if you want to describe a volume container in your figfile, you must expect it to run. The best you can do is

data:
    image: tianon/true
    volumes:
        - /my/volume
service:
    image: whatever
    volumes_from:
        - data

But it shouldn't be necessary to run a DVC at all. In the commandline Docker client I can do:

c=$(docker create --entrypoint=_ -v /my/volume scratch)
docker run --rm --volumes-from="$c" busybox dd if=/dev/urandom of=/my/volume/foo bs=1M count=1
docker run --rm --volumes-from="$c" busybox du -hs /my/volume/foo
docker rm -v "$c"

It's admittedly a little strange to have to provide an entrypoint to scratch, but it doesn't ever run, so it doesn't make a difference. It would be nice to be able to do this in Compose as well.

data:
    image: scratch
    create_only: true
    volume: …

Related:

• #697 asks to be able to run a subset of containers. This is similar in that it asks not to run a container. Perhaps this provides a use case for that, but that issue still implies that the containers can be run. This proposal is to mark containers for creation, but never to run them.
• #613 - Despite the similar title this is a feature request with significantly different implications.

[immortal-tofu]

+1 for this feature

[MrMMorris]

+1

[aanand]

I think this would be valuable, especially for data-only containers. To bikeshed: I don't like create_only: true. Perhaps start: false?

[Vrakfall]

I find start: false not very self-explaining. What about data_container: true or data: true. Even data_only: true would fit.

[MrMMorris]

@Vrakfall I would say that data_container, data_only and especially data are even less self explanatory. There may be instances where you would want to create a container and not run it that isn't a data only container.

I +1 start: false as it is very easy to understand what that means. Maybe even running: false or something like stopped: true similar to the Ansible module syntax.

[kojiromike]

I was shooting for something to distinguish the concept of running from automatically running. Maybe that's unnecessary. start: false seems OK.

[dnephin]

I am still really skeptical of any new config setting for this. I think we can solve these issues in a better way. For one, you can already do this by just listing the services you want to operate on. fig up -d <servicea> <serviceb>

I don't quite understand the problem with having a data volume container run echo or true. It seems like that isn't going to really harm anything, and it lets us keep things simple. fwiw, I use data volume containers extensively, and I just use CMD echo.

I think #988 is a better way to address the "administrative container that can run, but shouldn't run by default" issue. You can put those services into separate files, and just include the linked services.

[MrMMorris]

@dnephin that is a fair suggestion, but it will not fit all use cases such as:

• image used for db has an entrypoint set that doesn't allow for it to run echo or true
• I believe that a container running true/echo would be started again when compose is run unless you run with --no-recreate?

Let me know if I'm misunderstanding

[aanand]

@MrMMorris Regarding the first point: If we remove the need for intermediate containers - either by using renaming (#874) or manually copying volume config (#858) - then the only requirement for the image used is that it can run _a command of some kind_. I don't think that's a huge issue?

Regarding the second: does it matter that the container is started again when Compose is run?

[MrMMorris]

@aanand just so I understand: does https://github.com/docker/compose/issues/874 and https://github.com/docker/compose/pull/858 mean that you won't have to use the same container for the data as you do for the db as pointed out by @cpuguy83? http://www.tech-d.net/2014/11/18/data-only-container-madness/

as for the second point, I actually didn't consider whether or not the container starting again would be an issue or not. More of a note :+1:

[aanand]

@dnephin I'm surprised that containers with CMD echo work, given that docker-compose up is currently supposed to error out if a container exits.

Does it only work with -d? If it works without -d, is it only because of the race condition fixed by #738? If so, we should merge that and then address #741.

[dnephin]

Yes, true. It only works with -d, which is where #741 came from.

[kojiromike]

@dnephin I don't think your suggestion to use fig up -d <servicea> <serviceb> applies. If I have the following figfile:

svc1:
    image: scratch
    volumes:
        - /foo
svc2:
    image: tianon/true
    volumes_from:
        - svc1

Then even if I explicitly tell fig only to run svc2 with fig up -d svc2, it will still try to run svc1 because it is a dep of svc2. So how does specifying svc2 help in this case?

[dnephin]

I'm pretty sure that up only considers links as dependencies. volumes_from is only considered for building of containers, so doing docker-compose up -d svc2 should only start the one container (and any linked containers).

[kojiromike]

It's possible that there are semantics for _dependency_ that I'm abusing here. Regardless, fig 1.0.1 and docker-compose 1.1.0 (1.1.0-6-g44f5a3e) both definitely try to start the volumes_from container when calling the service that is supposed to use that volume.

08:16 ~/docker/test $ fig --version
fig 1.0.1
08:16 ~/docker/test $ cat fig.yml 
svc1:
    image: scratch
    volumes:
        - /foo
svc2:
    image: tianon/true
    volumes_from:
        - svc1
svc3:
    image: tianon/true
08:16 ~/docker/test $ fig ps
Name   Command   State   Ports 
------------------------------
08:16 ~/docker/test $ fig up -d svc3
Creating test_svc3_1...
08:17 ~/docker/test $ fig up -d svc2
Creating test_svc2_1...
No command specified

You can see from the above that the "No command specified" complaint is actually due to svc1 being used as a dvc.

[dnephin]

The up command you ran is missing from that output.

I just tried it out with a similar example. I ran docker-compose up -d web on a container that has a volumes-from a data volume container. The data volume container shows in the ps, but if you inspect the container, it's state has a 0 timestamp for startedAt (meaning it never ran). If I run docker-compose up -d web again it recreates the web container, but doesn't touch the data volume container.

I used tianon/true for the data volume container and never ran into any errors.

[MrMMorris]

@dnephin so for now, can I have my command for the data only container be something like CMD echo and I can be sure it won't be restarted with -d?

[aanand]

If Compose isn't starting volumes_from dependencies, that seems like an accidental feature - we should make sure that's the behaviour we want and test/document it.

@MrMMorris: if the data-only container has echo as its command, what's wrong with restarting it?

[MrMMorris]

@aanand it's not that there is something 'wrong' with it restarting, I would just want to know whether I should be expecting it to or not.

I answered my own question anyways, and it doesn't restart if I up -d a specific service. Only if I up -d everything.

[munhitsu]

+1 running/start: false

[rqc]

first of all :+1: for this effort. I am trying to orchestrate a solution that requires to create a d.ata container using docker-compose. I think the naming of the feature within the yml language should be similar to the docker commands to make it as close as a one-to-one mapping. Like the docker team have said in the past, docker in general should be a consistent API, even if you go from the CLI to swarm or in this case to docker-compose. So, perhaps something in the likes of create or create-only would make more sense.

Thanks!

[sherter]

I would like to mention another use case here that doesn't even need a container to be created:

# typical LEMP configuration
nginx:
  build: ./nginx/
  links:
  - php
  ports:
  - "8080:80"
  volumes_from:
  - data

php:
  build: ./php/
  links:
  - mysql
  volumes_from:
  - data


mysql:
  build: ./mysql/
  environment:
  - MYSQL_ROOT_PASSWORD
  - MYSQL_DATABASE=joomla
  - MYSQL_USER=joomla
  - MYSQL_PASSWORD

# data only container: should be created, but not run
data:
  image: busybox
  volumes:
  - /usr/share/nginx/html

# containers for manual interaction (docker-compose run backup):
# should neither be created nor run
backup:
   build: ./backup/
   links:
   - mysql
   volumes_from:
   - data
   volumes:
   - ./:/backup

Maybe we need two options like start: false and create: false?

[d3vil-st]

+1

[ReSearchITEng]

+1
Did anyone find a work-around in the meanwhile?

[d3vil-st]

Yes, I manually create container with same name as compose, and run "docker-compose up --no-recreate"

[hadim]

+1

[schmunk42]

@ReSearchITEng My workaround is to use command: tail -f /dev/null to keep a container running

Setting something like command: none could also do the job.

[pikeas]

Bump, any movement on this? How about run: true/false defaulting to true?

[kojiromike]

This might seem crazy, but what if docker/compose didn't consider it an error if a service doesn't have an entrypoint at all? What if the lack of an entrypoint is just an indication that the container can exist without running? For images that don't have a native entrypoint you shouldn't have to say anything:

svc:
    image: scratch
    volumes:
        - /glorp/blatz

For images that do, you would need to be able to nullify it.

svc:
    image: mysql:5.6
    entrypoint: null
    volumes:
        - /flue/barge

I recognize this would probably require changes to Docker itself, but thought it was relevant enough to this issue to bounce it here, first. Thoughts?

[cpuguy83]

What if data containers went away completely at some point now that docker 1.7 supports named volumes and volume drivers?

Also no, you can't have a null entrypoint, this will not change in Docker.
But what could be done is flagging a service as a sort of temporary service basically signaling compose to not care if it exits..., or make "up" not consider a container that stops an error unless it has a non-zero exit code.

[aanand]

We've talked a bit about making docker-compose up not exit until _all_ containers have exited, which I suspect is what people want most of the time, and would also solve this problem.

[aanand]

See discussion in #741.

[dnephin]

I believe this is fixed by #1754. If you disagree, please feel free to re-open for further discussion.

[tomfotherby]

I would find a start: false directive very useful. My use-case is that I'd like to define a container than can run our unit tests within my docker-compose.yml and I don't want it it run on docker-compose up - I want it to run with docker-compose run only.

[bfirsh]

Being tracked here: #1896

[casertap]

So... how can I create a data-container only using docker-compose.yml?
I could not find any answer.

[dnephin]

Use still need a command, so use something like tianon/true

For now you run up with -d, in 1.5.0 that restricting is removed.

[zx1986]

image used for db has an entrypoint set that doesn't allow for it to run echo or true
I believe that a container running true/echo would be started again when compose is run unless you run with --no-recreate?

@MrMMorris you totally made my points!
@sherter thanks for your using case, I learn a lot!
@d3vil-st thanks! a good tmp solution for me.
I run into this problem today, and I'm looking for a good practice.

[divramod]

+1

[zored]

+1

[achembarpu]

Hi,

So, I've gone through this and multiple other linked-to-and-from issues. I'm confused about whether there's a working solution or not...

Basically, I want to make node_modules and bower_components persistent.

Here's my current setup:

docker-compose.yml :

elasticsearch:
    image: elasticsearch
    ports:
        - "9200:9200"

kibana:
    image: kibana
    ports:
        - "5601:5601"
    links:
        - elasticsearch

api:
    build: ./api
    working_dir: /app
    command: forever app.js
    environment:
        - NODE_ENV=development
    volumes:
        - "./api:/app"
    ports:
        - "1337:1337"
    links:
        - elasticsearch

dashboard:
    build: ./dashboard
    working_dir: /app
    command: forever server.js
    environment:
        - NODE_ENV=development
    volumes:
        - "./dashboard:/app"
    ports:
        - "8080:8080"
    links:
        - api

api/Dockerfile :

FROM node:4

RUN mkdir /app

WORKDIR /app

RUN npm install -g forever

ADD package.json ./

RUN npm install

dashboard/Dockerfile :

FROM node:4

RUN mkdir /app

WORKDIR /app

RUN npm install -g forever bower

ADD package.json ./

RUN npm install

ADD bower.json ./

RUN bower install --allow-root

Any ideas?

[quinncomendant]

This issue may not receive much attention because it has been suggested that Data Volume Containers don't provide much advantage over the use of Named Volumes (the use and creation of which can easily be specified in a compose file).

[nioncode]

There is no way to omit the absolute path to the data inside the container if we don't use data-only containers, or is there?

Example with data-only container:

version: '2'
services:
    app:
        build: ./app
        volumes_from:
            - app-data
    app-data:
        build: ./app
        command: echo 'Data-only container'

The app/Dockerfile declares which directories are VOLUMEs, so we don't need to set any concrete paths.

Example with named volume:

version: '2'
services:
    app:
        build: ./app
        volumes:
            - app-data1:/var/data1
            - app-data1:/var/data2

Here I explicitly need to create volumes for each directory that is listed as VOLUME in the chain of Dockerfiles that are inherited by app/Dockerfile. This is not so nice from a usage and maintenance perspective.

[ekkis]

I hate to add +1 posts (there should be a voting mechanism on github) but my use case is: I want to build an image with docker-compose but for creating containers I have another process that is based on user interaction. so I would like to see start: false implemented (or some other such thing that indicates I don't want a container created, only the image built)

actually, start: false is a bit ambiguous because it would indicate only that a container shouldn't be run, not that it should also not be created, so maybe:

create: false
start: false

or perhaps:

process:
  - build
  - create
  - start

though that can lead to choices that could cause errors like failing to specify build but asking to create (which would break if the image has not already been built)

[tn-osimis]

@ekkis If this doesn't pan out you might be able to use something like shipwright.

[ekkis]

@tn-osimis: thanks for the recommendation. in my case I don't keep my source on github, nor do I publish to the Docker hub