Compose: Add `copy` to the yaml configuration

[riquito]

I want to deploy different services using the same image but with a different configuration file.
Currently to achieve that I can

• build as many images that inherit from a common image plus a different COPY in each
• mount a single file volume

The first solution isn't feasible and the second it's overkill. I just want to copy a file, not keep it in sync

Proposed configuration:

myservice:
    image: foo/bar
    copy:
        - src:dest

[prologic]

What is the semantics of copy here? Are we copying files from the host where docker-compose is run from into the newly running container using docker cp?

[prologic]

AFAIK there is _no_ current way of doing this:

See:

``` #!bash
$ docker cp -h

Usage: docker cp [OPTIONS] CONTAINER:PATH HOSTDIR|-

Copy files/folders from a PATH on the container to a HOSTDIR on the host
running the command. Use '-' to write the data as a tar file to STDOUT.

--help=false Print usage
```

[riquito]

What is the semantics of copy here? Are we copying files from the host where docker-compose is run from into the newly running container using docker cp?

Yes

AFAIK there is no current way of doing this:

I see that the docker cp command is unable to do it. Yet it's still possible by accessing the container data on the host side (which I ammit is ugly)

http://stackoverflow.com/a/24805696/210090

[prologic]

Yeah I'm not going to advocate we do this at all. This will break a lot of things and assumptions about Docker. This is also highly dependent on the underlying storage driver being used as well. Docker is meant to be agnostic. I think we _should_ a different solution to your problem :)

[ljrittle]

Now that docker-1.8 added support for copy into a container via docker cp and swarm-0.4.0-rc2 added support for docker cp, it would be great to revisit this issue at the compose level. Here is the use case (which mirrors what we actually do in _almost_ production now):

A docker-compose.yml file that mentions many containers by CI-built image tag (i.e. we don't use build: in production; maybe we could now but it didn't play nice with swarm in past releases) that all need e.g. static hadoop configuration files mapped in which exactly match the deployment environment. Currently, one must manually sync the directory in which docker-compose.yml resides to the exact path on _each_ target docker-machine in the swarm. Then one adds a stack of volume: lines.

Supporting docker cp would allow the removal of a custom configuration file sync in our system deployment method and allow better control of when configuration changes are injected (since anyone changing the files mentioned in the volume: lines are affecting all prior deployed containers (and whatever time the internal implementations re-read said configuration, perhaps only upon the next container restart, perhaps when various tools inside the container start; which may be unexpected) and future (which is expected).

OTOH, (as mentioned briefly above) maybe we should use build:. The downside here is that we need to write an extra Dockerfile per deployment container type. One for the CI-built images and a second for the static configuration file injector. copy: in compose backed by the recent changes to docker cp would neatly allow us to avoid all this duplication.

[dnephin]

It still seems like using a volume is a better way to do this. It doesn't have to be a host volume, it should work with a data volume container.

[prologic]

Bedies docker-machine has docker-machine scp anyway which you can do with the preparation of the machine; and then you can do host volumes the normal way eith docker or docker-compose

[ljrittle]

"It still seems like using a volume is a better way to do this. It doesn't have to be a host volume, it should work with a data volume container."
I fail to see how that is best when the target is a swarm. AFAICS, one has no way to express that a data volume container should exist at every possible swarm node.

[prologic]

Can't you deploy a data volume container on very node via swarm itself anyway like you can with any other container ala scheduling strategy?

[ljrittle]

"Bedies docker-machine has docker-machine scp anyway which you can do with the preparation of the machine; and then you can do host volumes the normal way eith docker or docker-compose"
I currently do this. It is painful. People managing deployments forget to to move the configuration files. If it was supported in docker-compose yml then it would happen whenever a container was recreated without the manual step(s).

[ljrittle]

"Can't you deploy a data volume container on very node via swarm itself anyway like you can with any other container ala scheduling strategy?"
Maybe we could but I admit that I don't see how in light of scaling.

[ljrittle]

"Can't you deploy a data volume container on very node via swarm itself anyway like you can with any other container ala scheduling strategy?"
Humm, if I know that my swarm is size 10 nodes; can I just scale the data volume container to be size 10 with a policy that no dups are allowed on a given docker engine node? The problem (unless I have missed something) is that e.g. containers that need to reference that data volume have no way to --volume-from a container with an index that matches. Guys, I have been looking at this problem for 2 months now. The best I could do is create a script that uses docker-machine scp. But it is a manual step upon after editing a configuration file and before docker-compose up. It also requires that one can write the exact path on the target machines of the swarm as the root of the docker-compose.yml (i.e. it smells like a major hack).

[prologic]

Perhaps my efforts in factory can help automate this process of spinning up docker machines and swarm clsuters with the "right" data sitting there ready to use? (_albeit it's still in early development and I'm using it to combine docker-machine and docker-compose in a single setp_0.

[dnephin]

copy is an operation, not part of the service configuration, so it doesn't belong in the compose file.

Docker 1.9 is adding a new volumes api, and volume drivers are in already. Volumes are really the correct way to support this. Not trying to copy files around.

Thanks for the suggestion! but I don't think this really fits with the design of compose right now.

[dutgcom]

copy is an operation, not part of the service configuration, so it doesn't belong in the compose file.

What if one want to copy just a configuration file? (to a docker swarm, so volume_from wouldn't work)

I have exactly the same situation, want to copy custom mysql, nginx, haproxy etc. config files, we have clusters in production, and instead of just using one copy command to copy configs from local to remote

I have to use images, build, pull each time, while there are no need whatsoever and just copy command will save lot of deployment and development time

[johnculviner]

This is very surprising to me that more people don't have a problem with this not being readily scriptable. Am I thinking about docker wrong? I think my solution now would be to use ansible to copy out the config files to the host and then volume mount from the host to the container(s) that need it

[unleashed]

Hitting the same issue. I am reusing a Dockerfile for N different services and I don't want to create N different Dockerfiles with their own COPY command or create a volume just for that.

[dutgcom]

Btw I'm currently solving that by running dozens of lines of sed commands - http://unix.stackexchange.com/questions/112023/how-can-i-replace-a-string-in-a-files

Maybe not optimal solution but works considering that docker-compose doesn't support copying files.

Btw mounting and copying of course possible but it goes against automatization docker-compose offers - if I would prefer to do everything manually I wouldn't use docker-compose, wouldn't I?

[johnculviner]

The more I've dug into Docker the more it seems like a lot of this tooling is good for "hello world"/"look how fast I can spin up this cool thing with\ shell\ commands\ like\ this\" on a single machine type use cases (docker-compose included). If it gets much beyond that things start to become quite complicated and IMO fall apart a bit which is where traditional provisioning tools can come in and save the day when used appropriately.

I've found using Ansible to set up machines in an idempotent way to be very easy when I have machine specific volumes, files and templated config files that must exist for the docker image to start. Then at the end I can use Ansible to invoke docker-compse or even use the Ansible docker module whose syntax is nearly the exact same as docker compose with some nice added bonus features.

At the end of the day the above approach allows everything to be scripted, idempotent and most importantly checked into source control.

[ColinHebert]

@dnephin some people above suggested the use case of CP being part of the configuration, would it make sense to revisit the assumption that CP is only for operations and not for configuration.

A simple example is creating a container, copying the configuration across in the correct place, starting the container once the configuration has been copied.

This allows the creation of images that do not embed the actual configuration and only deploy the configuration as the container gets started.

[dmikov]

Definitely will be very helpful. I am working with postgres image as database back end now and I don't want to rebuild it, just to update a script in /docker-entrypoint-initdb.d folder.

[ctindel]

Why don't you guys re-open this issue until the feature is implemented or a workable solution is found that people like, since nobody here likes the suggested options?

[NicklasWallgren]

This issue should be re-opened, as @ctindel mentioned.

[metmajer]

+1 for having the issue reopened

[lhridley]

+1 for opening as well.

[ghalusa]

+1 for reopening.

[dixiedream]

+1 for reopening.

[pbrueske]

+1 for reopening.

[AnthonyWC]

+1 for reopening.

[GeekHare]

+1 for reopening.

[BorzdeG]

+1 for reopening.

[michaelarnauts]

What's wrong with a read-only volume?

myservice:
    image: foo/bar
    volume:
        - src:dest:ro

[koooge]

@michaelarnauts see this.

[tonivj5]

👍 for reopening

[nguyenphuocpham]

+1 for reopening.

[andresgaragiola]

+1 for reopening.

[Orfen-0]

+1 for reopening

[ericlery]

+1 for reopening

[jayaskren]

+1 for reopening

[matthewh]

+1 for reopening

[legovaer]

+1 for reopening

[QuackMartins]

+1 for reopening

[jasonterando]

Here's my rationale for allowing a copy in Compose.

I have an application that has runs on numerous Containers. For sake of argument, let's stipulate they all run Apache. They all have DocumentRoot definitions pointing to things like "/var/www/partX". The Dockerfile knows nothing about the contents of the Apache configuration, and the Compose defines the volume reference for /var/www/partX. This works great for development, because I can debug and tweak code without modifying container contents.

The rub is when I want to deploy. One of the main attractions about Docker is that I can deploy a container that is its own universe. In order to get everything in the container, I have to copy the files, which means that I would have to have separate Dockerfile and Compose definitions from what I use for development. On a large project, this means I have to maintain two sets of definition files, which introduces more possibility for errors.

My preference would be to maintain a single set of Dockerfiles, and do the orchestration on the Compose file. Compose would be where I define if I am going to set up /var/www/partX as a shared volume or if I am copying files into the container.

Deploying with shared volumes seems like a can of worms. If I am updating an existing version in production that depends on a shared volume, I can't update that shared volume without setting up a maintenance window or coming up with a scheme for versioning my shared volume. I am tightly coupling containers with external files, and if I'm introducing that type of complexity, I am negating some of the benefit of using Docker.

It seems like I could script changes to the Dockerfile definitions to conditionally copy files, but it seems more "logical" do handle all of my Docker logic in the Docker universe, and Compose seems like a logical place to do it.

[ugurcanlacin]

+1 for reopening.

[kalinchuk]

+1. Would be very helpful to copy a configuration file in docker-compose.yml file instead of Dockerfile.

[DimYurich]

+1. Just ran into it trying to supply a config file for the service. Read-only volume seems like an option, but would rather copy it.

[raszi]

Simply mounting a volume could do the trick:

version: '2'
services:
  foobar:
    image: 'postgres:9.6-alpine'
    volumes:
      - './docker/schemas/foobar:/docker-entrypoint-initdb.d'

[jasonterando]

@raszi - Yah, the mounted configuration is the direction we're going to go for now. One of the things that I really liked about Docker though was the notion of a self-contained container that I could run integration tests on, without any concern for external dependencies (like linked configuration). If I am doing blue/green or A/B deployments, and there are differences in the configurations used by different versions of containers, things can get a little fragile. I can get around this by doing things like versioning configuration files in a shared volume of each of my environments, but it just seems more complex or brittle than it could be.

"Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment." To me, that ideally would include configuration. Compose is sort of doing a "build/compile and run", I think some of us here are wishing Compose could do a "build/compile, link and run". It is not how it works today, and I can understand the lack of enthusiasm for going that route, but hey, it's a forum, it doesn't hurt to ask ;)

[raszi]

I agree, it would be great to have this feature. I just posted my solution to help others.

[MeGaPk]

+1 for reopening.

[axklim]

+1

[burningalchemist]

+1 for reopening or different solution without volumes.

Currently using Environment variables in docker-compose.yml and j2 in docker-entrypoint.sh for parsing them to configuration files before running the main app of the container. Works for me, though it's a custom image and it can't be used out of the box with already available complete images.

[budzek]

+1

[patrickjm]

+1

[b0laj1]

+1

[mizanRahman]

+1

[oiwn]

+1

[Snake-Tn]

+1

[nubitech]

+1

[storytime]

+1

[mmontossi]

+1

[jeroiraz]

+1 for reopening

To add the possibility to have a different yaml file for each container instead of a config based on environment variables

[ecoulon]

+1 very needed feature

[damianvalderrama-olx]

+1 , nice to have feature!

[andreymaznyak]

+1

[vlean]

+1

[jjortega]

+1

[konstan]

+1

[hieudang9]

+1 reopening. Bookmarked

[shatil]

+1 this is a much-needed feature.

If there was some way to mark a shared volume as a layer, so the volume remained RO but the writes applied to a layer atop, I'd live happily without a copy directive. I'd like to share, for example, an extracted build artifact, but the service writes to that directory (logs or PID file, etc.).

[christabone]

+1

[MichalKalita]

+1

[omani]

+1

[OlegVorobyov]

+1

[jabberwockyS]

+1

[Julien00859]

+1

[kolyaflash]

+1

[ikrauchanka]

+1

[che]

+1

[ghost]

+1 For reopening as it would be a very useful feature

[Julien00859]

Maybe the template feature of https://github.com/jwilder/dockerize can help some of you

[BlueHotDog]

+1

[hubun]

+1

[sylvain261]

+1

[tindleaj]

+1

[vegasbrianc]

+1

[zmellal]

+1 to reopen the issue

[leshcat]

+1

[zmellal]

Use docker secrets, in this example secrets are used with nginx image to copy server certificat and the config file
https://docs.docker.com/engine/swarm/secrets/#intermediate-example-use-secrets-with-a-nginx-service

[washtubs]

That's a good point. Honestly it seems that since we already have secrets, we should have copy, as secrets are, in a way, just a specific type of file that you would copy over. Namely, one which is more carefully provisioned, which we don't need for config files.

It beats host mounting a volume for every configuration file you want included.

[Oleffka]

As said before:

...
volumes:
- ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
...

[washtubs]

The reason copy is better, a) in swarm environments, where hosts are in fact separate machines, you should not have to have the exact same file in the same place on every machine. b) Copy can also be restricted to the folder compose is in, whereas a host mount requires an absolute path. c) I think the point of docker is to minimize the number of places that the contained application breaks into the host. Host mounts, even when configured as read-only, create a responsibility on the host to have and preserve files. We shouldn't have to use host mounts to solve this simple problem.

I think copying could be modeled similarly to the docker build process, where the Dockerfile is usually shipped in a git repository, along with all the files it needs to send to the build context. The Dockerfile isn't permitted to send anything that exists outside of it's directory structure, even if it's symlinked correctly. We could have a similar thing with compose, where src is restricted to the directory (and those below) of the docker-compose.yml. Basically, the workflow would be, docker-compose creates the container, and then for every src:dst pair, find the src under the current directory, and copy it into the unstarted container, then start the container.

Currently, avoiding host mounts can be achieved by adding additional Dockerfiles and building the conf files into amended images, but this seems like overkill to me, as it involves retagging a new image, or forcing users to use docker-compose build attribute instead of the more preferable (IMO) image attribute. If you're already using the build attribute in your service definition, and you want to do things this way, you are forced to pollute your otherwise agnostic image builder with a highly opinionated configuration file that should just belong to compose and the deployment process.

[paulwilton]

+1 for docker-compose copy feature

[and1990]

+1

[Xplouder]

+1

[kfrendrich]

+1

[rajeshkaushik]

+1 for copy.

[PMDubuc]

+1 for docker-compose copy feature. I think the arguments in favor above are compelling.

[ykalinov]

+1 for copy

[armadadean]

+1 for copy.

Although in my POV. It can become handy but it can also become out of control or abused for many users and can even make volume obsolete. If you just need files to be put onto the container especially for swarm then you can just use configs or secrets; But for my case like putting static folders in a nginx container, I have to use multi-stage builds in my Dockerfile to put those static folders in the container which takes a lot of process before my goal is achieved. So I think adding a copy option could be very useful

[spetrashov]

+1

[ekapratama93]

+1

[mrkvost]

+1

[zzart]

+1

[mracos]

+1

[AlecJY]

+1

[petr-stupka]

+1

[meigallodixital]

+1

[sajjaadalipour]

+1

[RW83]

+1

[conker84]

+1

[Nevario]

+1

[marenaso]

+1

[ryshi]

+1 for reopening

[rhk4360]

+1

[jonsecchis]

+1 for reopening

Wouldn't a feature that operates on a volume after the container (or itself) reachs certain state work?
It could be a flag like readonly, but copyonly instead! Then you would have the regular volumes operations pipeline/syntax with the added functionality of removing the link after a state of sync is reached inside the container...

[scottlindner]

+1 for reopening. Volume mounting is not always an option. Which is why I'm posting here today. I need to modify a large number of containers by hand to copy files in due to a unique scenario where volume mounting is forbidden.

[atheinzen]

+!

[virus-warnning]

+1

[ghost]

+1 for docker-compose copy feature

[afdevries]

+1

[ukbe-aurea]

+1

[stuaxo]

I'm starting to think this is pointless. Docker only add features around the particular use-case they built docker for - this is one of the features outside their use-case so they will probably never add it.

[delcon]

+1

[shin-]

@stuaxo Just because a lot of people are asking for footguns, doesn't mean we should automatically implement them.

In this case, files that are required by a service to run should either be baked into the build (declared in Dockerfile), made available through volumes, or (in the case of docker stack / Swarm mode) exist as config or secret objects. Copying files into potentially multiple destinations at runtime is slow, error-prone, and not failure-resilient.

[stuaxo]

Cheers, apologies if that came across as grumpy.
When you say "running a service" I guess you mean a long running thing like a web service, Docker has this covered well.

Dockers layering and caching are handy for other things ;

I put my old WordPress blog into docker, but only run-it sometimes for a few minutes so I can export data or check what a post looked like there.

This took a bit of work to use docker compose to separate out MySQL and Apache, but I wouldn't care if they where all smooged together.

Dockers caching is fun when experimenting with building desktop libraries like Cairo. Most of the things I was trying to do I've forgotten the details of, but are more on the building-something side of things rather than anything to do with running services.

[kingsman142]

+1 docker-compose copy needs to be added. It allows for much easier flow of configuration files and updates to containers.

[prologic]

Use docker cp and docker config

On Fri, Mar 2, 2018 at 21:19 James Hahn [e-mail] wrote:

+1 docker-compose copy needs to be added. It allows for much easier flow
of configuration files and updates to containers.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/1664#issuecomment-370055493,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOv-q5VO9n9HDvBP6YJ1BCePz5tGu-pks5tabdrgaJpZM4FTTPg
.

>

James Mills / prologic

E: [email protected]
W: prologic.shortcircuit.net.au

[chinmaya-n]

+1 for docker-compose copy. The idea is to have copyonly for files as described here

[38leinaD]

+1

[smaley07]

+1

[WebMatrixware]

In this case, files that are required by a service to run should either be baked into the build (declared in Dockerfile), made available through volumes, or (in the case of docker stack / Swarm mode) exist as config or secret objects. Copying files into potentially multiple destinations at runtime is slow, error-prone, and not failure-resilient.

Except in many cases all you want is to update a single config file, in which case baking your own image of a standard official image is bad practice and builds in lots of technical debt for down the road, volumes are overkill X10 and you do not want or need the files to keep syncing not to mention the problems of having to deal with the whole folder and not just a single file in a default location, and you need to update a system file not just get a config setting.

I understand your concerns, but just telling everyone their need is not real will only drive people away from your project and the community that wants to use it and support your work. We want to work with you here to find a solution that works, but all you want to say is that we don't need what we think we need.

[darkguy2008]

Wow I can't believe this hasn't been done yet. I thought that this basic functionality would've been in docker-compose by now. Oh, the beauty of OSS projects...

Solved it with docker-compose --force-recreate --rebuild, it copies the new files if they've changed (or so it seems). Not ideal but it works.

[zeeke]

+1

[elemir]

+1

[avgoeid]

+1

[umedjan]

+1

[ascourtas]

+1 this is low key driving me nuts, it seems silly to have to mount a volume just for one config file.

[Payden-Pringle]

+1

We have a way to add files to a container using docker compose with volumes. I've noticed that the Compose documentation does have a configs setting now, so we can add single files as well.

https://docs.docker.com/compose/compose-file/#configs

However, what I believe people want is a way to merge files from the host into a directory in the container with preference being overwrite if exists, the way you'd expect a copy to work.

It's strange to me that the volumes configuration in the compose file has a nocopy option:

https://docs.docker.com/compose/compose-file/#volumes

volumes:
  - type: volume
  source: mydata
  target: /data
  volume:
    nocopy: true

Which is described like this:
volume: configure additional volume options
nocopy: flag to disable copying of data from a container when a volume is created

So it disables exactly what is desired?

My desired use case is web applications for development. I want the image to contain everything it needs to serve a website including a default website to serve, but have an option to merge/overwrite those files with the current working state of a website without other configuration (the key) beyond having the files in the right place on the host. Files that, when copied to the container, are ephemeral. Meaning not in a network share.

An example of a framework that would benefit from that is something like Laravel. Where I make an image that will be capable of serving the Laravel splash page with no other configuration (volumes, network shares, etc), but the compose file is written to copy files from the host to the container's Laravel root so that they can be overwritten.

Using a network share for this is extra upkeep for an ephemeral environment (development isn't always constant). Using a volume mount for it, with a framework like Laravel, requires building the app in that volume which means installing PHP and composer on the host or sharing the location on the host the app would be created at. Both creating more maintenance.

Edit:

After testing, it seems that when you mount a volume, the files that existed in the image at that location are copied over to the persistent volume with preference to the volume's original contents. Meaning it doesn't overwrite.

I actually think this is a solution for the desired outcome.

[su-narthur]

Hey, guys, just FYI, it's better to give a thumb-up reaction to the top message on this issue than add ing a "+1" comment, as issues can be sorted by how many reactions they've received.

[sentinel09]

+1

[flavienbwk]

+1 Please Docker team, consider your community.

[masaeedu]

@shin- The problem right now is that the volumes feature makes it impossible to plug in a single config file from an external volume into a container.

Suppose I want to plug an nginx config file into /etc/something/whatever.conf without annihilating everything else that's in that folder. You can't create a named volume for a single file (see moby/moby#30310), and you also can't create a directory volume containing a single file, then mount that particular file into the container (see moby/moby#32582).

The only option you're left with is to pollute fixed locations in your actual Docker host with the config files and then bind mount them in. This of course goes to pieces if you want to move around between Docker hosts, or work with a Docker host you don't have filesystem access to (e.g. online CI), or even just run multiple stacks in parallel.

Something has to give way here, either the volumes feature in Docker needs to change so that we can actually use it to accomplish injecting config files into our stack, or we need to work around it in compose using something like a copy config option.

[shin-]

FWIW I'm no longer a Compose maintainer, but it seems to me that that use case would be best served by injecting the configuration file at build time using a Dockerfile.

[masaeedu]

@shin- Unfortunately the configuration files are not known at build time, the images are prepared on a CI server long before we know where we're going to be deploying them and in what configuration.

[WebMatrixware]

I'm with @masaeedu, I understand the conceptual ideal of keeping the function of compose from creeping and therefore not wanting to add this feature, but in real life the loss of this feature GREATLY reduces the circumstances where compose can be used without

A) having to cobble together some kind of of hideous script work around to stand up the containers in question (the very thing compose was meant to help standardize and keep us from having to do),

B) having to add some other tool that handles all of our config (which while workable adds a huge amount of overhead for small to mid-sized projects that really just need to copy a specific .conf and .env to each container, something you do not want to do at image build time, that is equivalent to saying that rather than your software having a config file you should hard code all of your options into the C++, it leads to dozens of extra images to have to maintain), or

C) having to maintain a separate image for every possible config of a given piece of software that we will ever need / build a custom image every time we have to make a minor config change....which is both unworkable in the real world of production software, and also makes it impossible to have the kind of robust QA and testing pipeline that we need for production software.