Moby: Deploy to swarm from multiple compose files

[Parnswir]

I'm currently testing the new --compose-file parameter for docker stack deploy in 1.13.0-rc6, but my use case requires deploying from multiple compose files (as possible with docker-compose -f base.yml -f additionalService.yml -f production.yml). At the moment, using globs or multiple --compose-file is not supported as far as I can see. Are there plans to support this in future releases?

A simple workaround/implementation strategy is to merge the compose files before deploying, but I would rather want to avoid using temporary files.

[vsellier]

I have exactly the same problem.

As a workaround, I'm using :
docker-compose -f compose1.yml -f compose2.yml bundle -o myfile.dab

It generates a stack file you can pass to docker stack :
docker stack deploy --bundle-file myfile.dab mystack

[Parnswir]

As far as I can see, (as of docker-compose 1.10.0) bundles do not support all features of the compose file format (network aliases, deployment, for example), which is why I use an external yaml merge tool as a workaround.

[bvis]

@Parnswir do you have an example of the usage of this tool?

[Parnswir]

I use merge-yaml, but I guess you can use any yaml library that follows the standard.
npm i -g merge-yaml
merge-yaml -i base.yml foo-*.yml -o /tmp/bundle.yml && docker stack deploy --compose-file=/tmp/bundle.yml deployment

[mahnunchik]

Any news?

[bvis]

This is not a problem for us anymore.

As a rule of thumb now we define the production configuration in the docker-compose.yml file. In case you need some variables you just need to add them with a default value for production.

Typically

services:
  my-service:
    build:
      context: .
    image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
...

And we have another file docker-compose.override.yml with development values (mount code volumes, etc).

This allows us to simplify things. When you are developing something locally you just do:

$ docker-compose up -d

When you want to deploy to your cluster, we have this command in our CI/CD pipeline:

$ docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth

[teohhanhui]

@bvis But that doesn't always work. What if we need some overrides in the production environment?

For example, ports mapping should never be in docker-compose.yml.

[bvis]

@teohhanhui I'm not sure I understand you, but what I mean is the the docker-compose.yml file contains your productive configuration, no more, no less. Imagine this file as something that is not useful for development and it just works in production, initially.

But.. what if my CI/CD pipeline requires an staging step? No problem! environment variables come to rescue!

From my example you can use more variables to define your stack behaviour, for example imagine that you need a different API endpoint when your app is being deployed to staging. You could save this with just this variable:

# docker-compose.yml
...
services:
  my-service:
    build:
      context: .
    image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
    environment:
      API_ENDPOINT: ${API_ENDPOINT:-https://production.my-api.com}
...

Remember: Your docker-compose.yml file should be deployable in production, then use as defaults your productive values.

First you build and push your images (the same images are used on ANY environment)

export MY_SERVICE_VERSION=1.2.3
docker-compose -f docker-compose.yml build
docker-compose -f docker-compose.yml push

But this would work in your staging environment (even development!) just defining an environment variable before launching your stack there:

export MY_SERVICE_VERSION=1.2.3
export API_ENDPOINT=http://staging.my-api.com
docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth

On other hand you need something for development, isn't it? You can define your environment variables adapted for DEV and you can use the docker-compose.override.yml for those things that are not just based on environment variables, like mounting volumes for the source code you are working on. It's as simple as redefine just for development:

# docker-compose.override.yml
...
services:
  my-service:
    ports: # This is needed for development!
      - 80:80
    environment:
      API_ENDPOINT: https://devel.my-api.com
    volumes:
      - ./:/project/src
...

And you can enjoy developing with you well known compose command:

docker-compose up -d

It will take care of merge your productive definition (docker-compose.yml) with your development configuration (docker-compose.override.yml).

I hope this more detailed description helps. If you have doubts or suggestions just tell me.

[teohhanhui]

Let me describe what I'm currently doing (with docker compose up -d):

In my project, docker-compose.override.yml is git-ignored. I provide dist files, for example docker-compose.override.prod.yml and docker-compose.override.dev.yml, which I then copy to docker-compose.override.yml and customize to my needs.

With the above in mind, this is the approach I've tried:

docker-compose config > docker-stack.prod.yml
docker stack deploy --compose-file docker-stack.prod.yml mystack

It seems to work well.

UPDATE: Starting from Docker Compose v1.12, you can do docker-compose config --resolve-image-digests to pin images. :tada:

[ph-One]

@teohhanhui I'm having issues with that approach -- using config to concat the files. It seems that config changes the ports section to use published/target syntax, versus "80:80" syntax, which stack does not seem to care for...

ports:
    - published: '12181'
      target: '2181'
    - published: '12888'
      target: '2888'
    - published: '13888'
      target: '3888'

services.{service-name-here}.ports.0 must be a string or number

[gligoran]

@ph-One What versions of docker and docker-compose are you using? Also what's the version specified in your docker-compose.yml file? Long version of ports syntax (which is what you have there) was added with docker-compose spec 3.2.

[vide]

@bvis I really like your clever hack and I will probably use the method you described, but still... this is a hack. Are there any news on this issue by any moby contributor?
It seems something quite critical to me, from an automate-deploy-to-prod point of view, while keeping the dev/staging envs as close as possible to the production environment.

[ph-One]

@gligoran I think you answered your own question, ha. version: '3.2'

[bvis]

@vide You can see it as a hack, but even if this task is finished I probably won't change my way because it gives me some advantages:

• I can use the "simple" command on each environment:
  DEV: docker-compose up -d
  NON DEV (in CI/CD or manually): docker stack deploy --with-registry-auth my-stack
• I can reuse the same commands between different products, I just need to change the stack name.
• I avoid duplicated configuration, I just need to change the things that are already different

It's just a question of preference and that, maybe, I haven't still found the use case that does not fit this approach.

[vide]

@bvis I didn't mean "hack" as a bad word in this context, but I'd rather see what's already implemented (and wildly used) in docker-compose present in Swarm stack deploy too, natively. I completely understand that your method will still work even when/if this is implemented, though.

[ph-One]

I always felt that DAB was a half baked idea, but it actually seems to work better than concatenating compose files through docker-compose config.
Far better luck doing

docker-compose -f ./docker-compose.yml -f ./docker-compose.dev.yml pull
docker-compose -f ./docker-compose.yml -f ./docker-compose.dev.yml bundle -o stack.dab
docker stack deploy --bundle-file ./stack.dab

[denisw]

This omission is annoying. The Docker Compose docs specifically suggest to create a base docker-compose.yml and additionally a docker-compose.prod.yml for production overrides. That this approach breaks down with docker stack deploy seems like a huge oversight.

[teohhanhui]

Why does it break down? It works pretty well for me: https://github.com/moby/moby/issues/30127#issuecomment-290456897

[denisw]

@teohhanhui, yes, this approach does work, thanks! You can even avoid the file copying by using the -f option:

docker-compose -f docker-compose.yml -f docker-compose.prod.yml config > docker-compose.stack.yml
docker stack deploy -c docker-compose.stack.yml

However, this approach is both a workaround and not documented in the relevant documentation sections. It was hard to discover to me (I had to look at and comment in this issue after all).

[denisw]

I created a pull request (https://github.com/moby/moby/pull/33290) to add the docker-compose config approach to the docker stack deploydocs.

[stephenlautier]

With the suggested example it's not working correctly e.g.

docker-compose -f docker-compose.yml -f docker-compose.prod.yml config > docker-compose.stack.yml

it "works" however it screws my paths from relative to be absolute for volume and build context etc...

e.g.

// from 
    build:
      context: ./Odin.Orleans.Sample.SiloHost

// to
    build:
      context: c:\dev\git\odin.orleans.sample\Odin.Orleans.Sample.SiloHost

Then when I get to use the image it doesn't work properly.

Am I missing something?

Using:
Docker version: 17.03.1-ce, build c6d412e
Compose file version: 3

[sulphur]

I think the safest way is to use ENV vars. For example when i try docker-compose -f docker-stack.yml -f docker-stack-staging.yml bundle -o staging.dab it ignores the secrets and has some problems parsing the tartget/published ports. My both .yml are "3.2"

or merge-yaml seems the safest for merging

[VolCh]

@bvis docker-compose.yml only for production only doesn't work if in the other environments we want exclude some keys. For example my current compose files fragments:

# docker-compose.yml - base for development and production
services:
  app:
    environment:
      - DB_HOST:app-db

# docker-compose.override.yml - for development
services:
  app-db:
    image: mysql



md5-f4ac6c732277bd0ee19c351c93b92b9f



# docker-compose.prod.yml - for production
services:
  app:
    extra_hosts:
      - app-db:10.1.0.11 #  'bare metal' production database IP

In the production we use 'bare metal' database, in the other environments - docker service. It's easy with docker-compose. If I move extra_hosts to docker-compose.yml for production swarm, I can't exclude it for development by any merge tools and I can get undefined behavior with extra_hosts and service name.

Also there is a risk to deploy dev stack without redefine default (production!) values and for example truncate production database and put dev fixtures in it.

[jmkgreen]

@bvis On the face of it, your comment https://github.com/moby/moby/issues/30127#issuecomment-290379763 sounds sensible. Until a new staffer joins and needs to use the same docker-compose.yaml but provide their own values for each variable.

This becomes a nightmare scenario. Someone updates the docker-compose.yaml to provide a reference to an external service defaulting it to the version in production. No-one checks that all instances of use get an updated set of environment variables and thus exists the potential for new developer to launch their environment and accidentally have a reference to the production environment still set.

Bundles sound like the intended solution, but the documentation as of today suggests otherwise. To quote:

and the concept of a bundle is not the emphasis for new releases going forward

Perhaps bundles are being replaced?

[ph-One]

I still am wary of using stack seriously until I can use multiple compose files (without a workaround!)

I will stick to the seemingly odd DAB until this is resolved. As others have mentioned, the workaround is not a solution when using compose 3.2

[waterscar]

It seems with the workaround using docker-compose config, the whole "deploy" section in the compose.yml file is ignored, which makes it impossible to schedule a service to a certain node. Any solution to this ?
Using docker-compose bundle result in a lot of scary "warning" stating that top level network and other stuff are not supported.
So far the only approach I can find is to manually create one docker-compose.yml file with docker deploy :(

[stephenlautier]

@waterscar thats because you're not using latest. After an update, that got fixed Im using the docker edge and it works fine.

I had opened a ticket regarding that

[japrogramer]

I like how docker compose -f base.yml -f overide.yml config > stack.yml works.
However there is one major draw back with the way it currently expands relative paths, or maybe i don't yet understand how mangers handle this.
Say that i have a swarm and i want to add a node as a manager, would i have to type out the command again so that the relative paths are correct?

[japrogramer]

@stephenlautier did you ever find a way to keep the relative paths?

[stephenlautier]

@japrogramer not entirely, no. However, even with full absolute paths, I don't get any issues when using docker stack deploy (can't remember how I used to get errors tbh).

So to answer the question above, you shouldn't need to fix paths when adding a new node as a manager (at least if you're using stacks) and it was already deploy. Generally, you would simply scale service

[vide]

I can confirm that using docker-compose -f config1.yml -f config2.yml config > final-config.yml works well and as docker stack deploy should work. Beware to update to latest docker-compose 1.14 otherwise older versions will produce an unusable config for more recent docker versions (>=17.05) due to, at least, an issue with the ports type (string vs integer).

[FredrikFolkesson]

Using docker-compose -f config1.yml -f config2.yml config > final-config.ym does not work if you in one of your configs have a $ that needs to be interpreted literally.

The standard variable substitution of using $$ does not work since the docker-compose config command will do the variable substitutions and your generated file will only have one $ which means that the stack command will fail since you are now no longer escaping the $

using 4 dollar signs aka $$$$ works, but that is really ugly.

[vide]

@FrederikNS thanks for the correction, I'm currently not using variable substitution in my compose files but maybe I'll do in the future

[ph-One]

I think the real underlying issue (if you can call it that) is that stack adopted the Compose file _format_, and we're (users) confusing it with adopted the Compose _way_ of doing things. It seems that every work-around listed here has some sort of issue. It's getting to the point of having to maintaining two separate sets of files (Compose files for Compose, and Compose files for stack), sooo...

At this point, I'd almost rather see a new docker-service.yml, or more work done to bundles -- something specific to running services. Taking a look at the Compose docs, it pains me to see, "The [insert option here] option is ignored when deploying a stack in swarm mode" littered about.

[VolCh]

@ph-One we use docker-compose itself for build, tags & push images. But we use dynamic (git branch, commit, etc) image's full name, so we need integration docker stack deploy with docker-compose build && docker-compose push. Now we basically have base docker-compose.yml with just service & image naming, docker-compose.build.yml with service's build keys and docker-compose.deploy.yml with service's deploy keys, networks, volumes etc. Also we have some docker-compose.deploy.{prod,stage,test}.override.yml, docker-compose.build.{prod,stage,test}.override.yml. For docker-compose we haven't problems, for docker deploy we need to use some tools (docker-compose config at the moment) for merging all .yml in one. It's ugly, but we have only one file for service-image binding at the both build/push and deploy phases.

[kmbulebu]

The docker-compose config method has bungled long format, tmpfs volumes for me. They were munged into an anonymous volume in the output configuration.

docker-compose version 1.14.0.

[ph-One]

@VolCh I'll have to revisit docker-compose config with the newest version of engine and compose. Previously what docker-compose config produced was not compat with the latest (at the time) docker stack deploy, but they may be more synchronized by now.

[kmbulebu]

We've been having pretty good success using spruce to merge the yaml files. Still a workaround, but it seems to actually work well. Oh, and it's golang.

https://github.com/geofffranks/spruce

[RehanSaeed]

Any update on when this is planned for implementation, it seems like a no-brainer that this should exist.

[mahnunchik]

What's wrong with you guys?

[zored]

If someone will develop this, add support of COMPOSE_FILE please. 😃
It would be amazing to set once this env with multiple files and then only call stack deploy.

P.S. Here is my current dev walk-around. Use COMPOSE_FILE or docker-compose -f:

docker stack deploy NAME --compose-file <(docker-compose config)

[vide]

@FredrikFolkesson I've just run into the literal $ issue but a a double $$ fixes it. So if you have a variable foo with value bar$ you should put in your compose file

foo: bar$$

and it will be interpreted as a single $ sign

[FrederikNS]

@vide wrong username tag

[thaJeztah]

Work in progress pull request here; https://github.com/docker/cli/issues/569

[HosseinAgha]

@thaJeztah does #569 land in the next release? It is 17.12 I suppose :)

[caleblloyd]

We've written a general purpose YAML/JSON merging tool to workaround this issue. It uses the same variable replacement syntax as both docker-compose and docker stack: https://github.com/boxboat/config-merge

config-merge is released on DockerHub at boxboat/config-merge. It has an added feature, which is the ability to patch the configuration with JSON Patch syntax, providing functionality such as deleting keys and inserting specific array indicies. Here is an example:

docker run --rm \
    -v "$(pwd)/test/docker-compose/:/home/node/" \
    boxboat/config-merge \
    local.env docker-compose.yml docker-compose-local.patch.yml docker-compose-local.yml \
| docker stack deploy -c - nginx-local

It's good to see that work is being done to add config merging into the Docker CLI. In practice, we have seen subtle differences in the way that docker-compose reads and validates configuration files versus the way that docker stack does. config-merge is great in these situations, as it allows for full manipulation of the configuration.

[marcellodesales]

@kmbulebu Love Spruce!!! Helps me merge yml and json file and helped automating docker stack!!!

[iraklisg]

@thaJeztah do we have any update regarding the status of https://github.com/docker/cli/pull/569 ???

[zizipoil]

+1, could we have this? this would really be helpful

[sulphur]

@zizipoil you can just use something like merge-yml :

merge-yaml -i docker-stack.yml docker-stack-$ENV.yml docker-stack-$ENV-app-versions.yml -o $ENV.yml

[zizipoil]

@sulphur I totally get there are workarounds but I'd love it to be a functionality of the tool itself.
I am totally with the comment of @RehanSaeed from 2017

A unified syntax (where possible) between docker swarm and docker compose would be amazing

[thaJeztah]

@zizipoil @sulphur this issue was closed because it has been implemented in https://github.com/docker/cli/pull/569 - if you have docker 18.03 or up, this feature is there

[zizipoil]

@thaJeztah thanks. I missed that. Great news

[ObjectMatrix]

"Note: Because support for stacks and bundles is in the experimental stage, you need to install an experimental build of Docker Engine to use it."

[thaJeztah]

@ObjectMatrix where did you copy that information from? because that's very outdated; "bundle files" remain experimental, but stack deploy using compose files has been a stable feature for more than a years

[ObjectMatrix]

@thaJeztah https://docs.docker.com/v17.12/compose/bundles/

[thaJeztah]

@objectmatrix thanks! that's a docs archive for Docker 17.12 - let me check if the current version still has that note; I guess the note needs an update (and was meant to refer to the bundle files being experimental)

[bpetersen]

Looks like you can specify multiple compose files: docker stack deploy --compose-file=docker-compose.yml --compose-file=docker-compose.prod.yml test-swarm. https://docs.docker.com/engine/reference/commandline/stack_deploy/#extended-description

[thaJeztah]

correct; see https://github.com/moby/moby/issues/30127#issuecomment-381152865