Moby: Add support for `extends` feature in Compose v3 / docker stack deploy

[shin-]

As can be seen in https://github.com/docker/compose/issues/4315 , the extends feature that exists in docker-compose seems to be popular among users despite its flaws. However, it has so far not been added in the Engine's implementation of the Compose format. So far, we have advised users to simply flatten their Compose file structure when using v3, but is this the long-term solution we want to go with? How can we provide a clear upgrade path for users who have come to rely on this feature?

cc @dnephin @vdemeester

[shouze]

I've added some notes https://github.com/docker/compose/issues/4315#issuecomment-280617251 to me bringing back extends as it exists up to docker compose file 2.1 version is not a good idea but the main feature I miss is to be able to declare abstract services (that should never be runned but could be used to group common properties of services for convenience).

[jakajancar]

Agreed on comments from docker/compose#4315 that the way extends worked was a bit spartan.

I won't recommend a solution, but FWIW, to show the extent of abuse, here are the conventions that adorn the top of our compose file:

#
# Docker Compose configuration
#
# Due to lack of "expressivity" in Compose, we define our own couple of service
# "pseudo-types":
#
#   - image-only services (name: *-image)
#
#     The only goal of these is to build images. No other services build images.
#
#     These have entrypoint overridden to exit immediately.
#
#   - base services (name: *-base)
#
#     These contain common configuration and are intended to be extended.
#
#     Their command (not entrypoint, to keep the original one) is overridden to
#     exit immediately. Service must support a command to exit immediately.
#
#   - task services (name: *-task)
#
#     These are intended for running one-off commands.
#
#     Their default command is overridden to exit immediately. Service must
#     support a command to exit immediately.
#
#   - "real" services
#
#     These are actual services that stay up and running.
#
version: '2'
services:
  ...

[nicodmf]

I think the extends as in v2.1 is a good choice. Extends is actually simple and understandable, this is too a good pratice for each environnement to have a little and readable transformation between dev, prod and env.

Actually, i have

• a common docker-compose file which present the base rules
• a dev docker compose extending it and with facilities for developper
• a staging docker compose extending common too and with facilities for this environnement
• a production docker compose extending common too and with facilities for this environnement, specially container replication, rules about restarting, etc...

This works, and i don't understand why we should search in this ticket a complete rewrite, but more the keep of an interessing feature. Extends is a good part with special use cases that other techniques can't resolv easily. I'm happy with extends possibilities.

[sambernet]

Blocking us from upgrading to v3.x format as well.

We keep our docker container definitions in a folder-per-instance layout, where each folder contains a docker-compose.yml that defines the env specifics for the container instance at hand. To factor out the common stuff (DRY) we use base service definitions in a parent folder and use extend.

So when I need to manage a particular container instance, I just need to cd into the right folder and can then directly run docker-compose commands without further configuration (no -f flags required that team members need to lookup or know, just works as expected out-of-the-box).

[SC7639]

I'm blocked from using version 3 compose files.

I use services.yml to define the base layout of my services and then extend that with dev.services.yml for my development evironment (mostly adding volumes) but I like the (DRY) re usability of extends when it was added. It's not a deal breaker but it would keep me from moving to version 3 unless there is a must have feature.

[sambernet]

I'm not against improving the v2.1 'extends' solution with something more appropriate though. Something like abstract services could be both more safe to use and more powerful.

For example (since it's abstract) I could imagine supporting abstract volumes/mountpoints/networks, where the abstract base service defines a local mount point or a required network for a service, without defining the host part - meaning a deriving service could then define/map the host part of the mounts/volumes and networks to whats appropriate in his host/stage environment.

That's an example of something we can't do now with 'extends' as far as I know, and would open some interesting possibilities.

[alwaysastudent]

Taking away extends feature was not helpful at all. We have a lot of web services started with the same volumes mapped, environment variables and labels set. They also have same healthcheck policies. And Not to mention ports. Combining compose files using multiple -f option is tedious and error prone if you are dealing with multiple projects in the same host.

[alwaysastudent]

@shin- will this be brought back on 3.2 schema ?

[efrecon]

I really, really would love to get extends back into the file format. I have been using it to further refine abstract services for a number of projects. I understand that multiple -f options fits almost the bill, but it doesn't always work. For example, I rather often change the name of the abstract service into something that is more meaningful in the context of the project at hand. This is something that the overriding that occurs with multiple -f options does not support.

I would not mind loosing the exact extends though, as long as their is some other way to "instantiate" an abstract service in a file.

[jazzfog]

I have a setup with a common service file and bunch of services extending it, changing mostly volumes, so I would say I rely on extends feature and do not see other good way to describe my setup.

--frustrated

[rationull]

+1
I can see the logic behind the recommendation to flatten docker-compose files but I don't like the idea of duplicating the code that defines our development service topology across multiple projects. We'll use the -f workaround for now, with shell scripts so that developers don't have to remember which services to include in which cases.

I'm hoping a satisfactory solution can be found here to enable better factoring of compose structures!

[MadBomber]

At the top of my use case list is to DRYup my configuration managed hoard of services. I thought I'd be able to take advantage of 'extends' if v3. I don't want to go back to v2... and I don't want to have special cases where I must use the -f process work-around.

[cirocosta]

Hey, did anyone start working on this? I can't find a PR to keep track. It'd simplify a lot some things for us here as well (use-case very similar to some described above).

[JanNash]

Since version 3 is frozen and I needed a way to share common configuration, I hacked a small workaround, which I think I could share here (I'm not sure if this is the right place but feel free to tell me where else I can share this info :))

I'm not going to elaborate on it here, since there's a readme in the repo.
Have a nice one, everyone ☮️

Edit:

Sorry, here's the link :D

[gustavosbarreto]

+1

[fabiohbarbosa]

+1

[ccamp46]

+1

[JanNash]

Thanks for the +1s 💃
In the meantime, I've found another docker noop image, which smaller by a factor of 10^3 (due to the actual noop being written in assembly).

Unfortunately, there's no License in that repo. I already wrote a message to the owner on facebk but he didn't answer yet. Maybe he'll add a license if more people query him about it :)

[grayside]

Something that might help some of the extends use cases (those within a single file) would be support for YAML anchors: https://learnxinyminutes.com/docs/yaml/

It appears that the JSON Schema may be failing validation on them service must be a mapping, not a NoneType..

[JanNash]

Hey, @grayside, yaml anchors do work, at least for me. See my comment above for how I use them.

[shouze]

Ok but it's too sad to use some noop service no?

Especially for env vars, what kind of values those env vars handle? If it's about secrets, use swarm secrets feature (or any other secrets solution). If it's about settings, we're ok that most of the time settings are app/service specific, not intended to be shared between services.

If you need to share settings between services, most of the time it's when you launch the same container image but for different runtime purposes/tasks (consumer, producer, http worker, ...).

[JanNash]

If it's about settings, we're ok that most of the time settings are app/service specific, not intended to be shared between services.

I tend to disagree. In the project I'm currently working on, I use it for example for volumes:

# Volume paths
environment:
  - &volume_a        volume-a:/usr/share/my_project/volumes/volume-a
  - &volume_b        volume-b:/usr/share/my_project/volumes/volume-b
  - &volume_c        volume-c:/usr/share/my_project/volumes/volume-c
  - &volume_d        volume-d:/usr/share/my_project/volumes/volume-d

Now I can specify these volumes like that:

volumes:
  - volume-a:
  - volume-b:
  - volume-c:
  - volume-d:

services:
  some-service:
    image: some-image
    volumes:
      - *volume_a
      - *volume_b

  some-other-service:
    image: some-other-image
    volumes:
      - *volume_b
      - *volume_c

  some-third-service:
    image: yet-another-image
    volumes:
      - *volume_a
      - *volume_b
      - *volume_c
      - *volume_d

This makes it way easier to navigate different volumes without having to think about which container you're in. Imho, this is one way to make your docker-compose setup more consistent and easier to use and maintain.

[shouze]

Ok yes I understand @JanNash but in your example below you don't have any noop service right?

[Yajo]

But anchors are not enough for many cases.

My case involves supporting several environments for the same project. You can see a scaffolding for our projects here.

When you develop, you use devel.yaml, but in production you use prod.yaml. There's also test.yaml. All of them inherit from common.yaml and get some common variables from an .env file.

Each one has its own peculiarities:

• devel environment aims for development speed, so it uses build args that produce faster builds, mounts volumes with the app code from the developer computer, and adds some dummy services that isolate the app from the outside world.
• prod aims for stability instead, so it downloads and compiles all code, and bundles it in the image itself instead of using volumes. Builds are slower, but more secure.
• test aims to be exactly like prod, but with isolation from external services, to avoid polluting the external world.

This simple separation allows to have a very agile and flexible DevOps pipeline, where everybody uses the same code among different stages, with just little tweaks depending on the environment in use.

I tried to move to compose file format v3, but not only extends is unsupported, but also .env, so right now it would be a maintenance nightmare (more because of the lack of .env, to be honest). When deciding between Swarm and DRY, we chose DRY for now, but some day we'll need Swarm and I hope that day both features are supported again... ☺️

... or at least we have a way to generate a valid DRY-less format from a DRY-ful solution. I thought docker-compose bundle was for that, but seems to be doomed to deprecation right now...

... or we have a different tool that makes everything (I'm keeping an eye on ansible-container too). But surely this is not "the fix".

[grayside]

The link in https://github.com/moby/moby/issues/31101#issuecomment-301212524 includes a README with the working example of YAML anchors. Looking it over and trying it again today, it works fine. Not sure what I'm doing differently.

@JanNash 👍

[JanNash]

@Yajo I hear you and as said, it's a workaround and it would be better by an order of magnitude if there was a good, built-in, DRY solution supplied by docker/moby/docker-compose (whatever is the correct reference). Let's all hope that will come soon because apart from that, I'm pretty happy with docker compose 👍

~For the sake of missing .env support, I also hacked a workaround (my project's not in production yet, so my talk is a bit cheap, I know :)). To support different sets of env vars (dependency and image versions/tags, for example) in different environments (for me right now, that is local-development and a small dev-server), I use two files, local.env and development.env and instead of running my commands by just docker-compose <command>, I either source the respective .env file into my shell before, or I run it like this: (. local.env && docker-compose <command>). Still a hack, but for now, I'm quite happy with this.~

Have a nice one, y'all 🚶

[JanNash]

Maybe even two orders of magnitude :D

[Puneeth-n]

@JanNash wait! is .env not supported anymore in 3?

[JanNash]

I actually don't know, I just read in a comment that it wasn't.
I've been using the local.env and development.env procedure mainly because I didn't know about autoenv when I implemented it :D
Sorry for possible confusion.

[JanNash]

Ah, @Yajo mentioned missing .env support in this comment.
Could you elaborate, @Yajo?

[Yajo]

Oh sorry, my fault. It's not that it is not working, it's just that you must specify it with env_file: .env, instead of being autodetected as before. Let's return to the original issue.

[bobbytables]

Is the discussion to drop extends anywhere? I'd love to read through it before giving our use case because I think it's well understood that it was a pretty used feature.

[tahonaPL]

Hi, I have one question - when? When "extends" support will be back in v3 ?

[eigood]

@JanNash you can get way super smaller than that. I just filed an issue in github against your repo, I got noop down to 1200 bytes from 750k on my machine.

[omerh]

I see that I can't use extend at the moment in swarm.
any ideas how to launch the same service with the same publish ports, and with one container on the service that has 1 extra environment variable?

+1 for extends support in swarm stack deploy

[acombe]

Hi,
We are running a microservices application that is spread in multiple git repositories (each one having its docker-compose file).
The deployment is leaded by one "root" docker-compose file that extends each services : for us this extends feature is really needed for stack deploy.
So also +1 to extends support in swarm stack deploy
Thanks.

[quolpr]

You can use YAML simple inheritance(see &default, <<: *default) as temporary solution:

version: '3'
services:
  worker: &default
    build: .
    command: bundle exec rake jobs:work
    env_file:
      - .env
    volumes:
      - .:/app
    depends_on:
      - db
      - redis
    links:
      - db:postgres
  web:
    <<: *default
    command: bundle exec puma -C config/puma.rb -p 3000
    ports:
      - "3000:3000"
  spring:
    <<: *default
    command: bundle exec spring server

Of course, extends feature is better

[Puneeth-n]

How about when you extend a different file?

[quolpr]

Yaml doesn't have file extending feature :(

[ABitMoreDepth]

Is there any update on this feature from a contributor to Docker? Is this planned to be re-introduced? If not, are there plans for something similar? If not, why not..?

[laugimethods]

@quolpr, I'm afraid your "YAML simple inheritance" code doesn't replace extends in most cases since the "prototype" (i.e. &default) will always be interpreted by Docker Compose as a service called worker. Which service a) needs therefore to be well defined, b) might be unwanted.

Anyway, definitely an interesting feature.

[quolpr]

@laugimethods you can also use YAML references:

version: '3'
services:
  spring:
    build: ./app
    command: /bin/sh -c "bundle exec spring server"
    volumes: &default_volumes
      - ./app:/app:delegated
      - bundle:/bundle:nocopy
  worker:
    build: ./app
    command: bundle exec sidekiq -v -C config/sidekiq.yml
    volumes: *default_volumes

(pay attention to &default_volumes and *default_volumes)

[quolpr]

But I really can't understand why extends feature was removed 🤔

[laugimethods]

FYI, to replace the missing "extends" feature, I'm now using a composition/merge of .yaml files:
https://github.com/Logimethods/smart-meter/blob/master/README.md#docker-compose

[mageddo]

extends works, is simply and mature, I think that if someone see extends as some anti-pattern then just don't use that, but please don't cut it off

[tiredpixel]

May I ask for a clear explanation of the intended approach without using extends? I use it extensively, particularly when inheriting from files contained within Git submodules, allowing a defintion of a meta-project handling cross-application networking wiring, etc. Although I'm well aware I can specify multiple docker-compose.yml files and have them overridden, does this mean that I would need to specify the interconnections at the command-line, rather than being able to check them into source-control using extends? Or have I missed some new feature somewhere in v3?

[teodorescuserban]

I am heavily using extends in multiple projects to inherit a common set of service attributes for different environment and different hosts (read: I use extends to inherit from a different file).

After reading in stupor the removal of extends keyword and trying to find a replacement that doesn't require daisy chaining -f docker compose files I am very curious what is the reason for removing extends.

I understand the problem with links and volume-from but just refraining for use them in base yml files seems the best thing to do.

It would be improbable to remove the wheel of a car just because it _might_ get used to turn the car upside down... right?

PS: noop and anchors looks interesting but it adds unnecessary complexity to the most simple projects...

As a very, very simple example:

common/common.yml

services:
  web:
    image: alpine:3.6
    build: .
    environment:
      DOMAIN:
      PREFIX:

dev/docker-compose.yml

services:
  web:
    extends: ../common/common.yml
    service: web
  ports:
    - "8080:8080"

prod/docker-compose.yml

services:
  web:
    extends: ../common/common.yml
    service: web
  image: the-prod-image:latest-release
  ports:
    - "80:80"
    - "80:443"
  environment:
    NEW_RELIC_KEY:

Just how you keep DRY principles without extends?

[mdotson]

Currently I see no reason to upgrade from version 2.1 due to this.

[dermeister0]

@teodorescuserban,

daisy chaining -f docker compose files

What's a problem with that? You may create your own scripts with short aliases to call docker-compose.

Use following structure:

common/common.yml

services:
  web:
    image: alpine:3.6
    build: .
    environment:
      DOMAIN:
      PREFIX:

dev/docker-compose.yml

services:
  web:
    ports:
      - "8080:8080"

prod/docker-compose.yml

services:
  web:
    image: the-prod-image:latest-release
    ports:
      - "80:80"
      - "80:443"
    environment:
      NEW_RELIC_KEY:

Commands

docker-compose -f common/common.yml -f dev/docker-compose.yml -p myproject up --build

docker-compose -f common/common.yml -f prod/docker-compose.yml -p myproject up --build

[Yajo]

I didn't know of that feature. Although it makes your CLI a 💩, it can work.

I think that if that's going to be the official replacement for extends, then there should be a way to make it easier.

For instance:

docker-compose.yml

version: "3"  # or whatever
extend:
  - ./common/common.yml
  - ./dev/docker-compose.yml
services: # Not required now
  # etc.

This way you can point to a single docker-compose.yml file that does all you need.

A useful alternative would be to support multiple compose files in the COMPOSE_FILE env var.

[gsong]

@Yajo

A useful alternative would be to support multiple compose files in the COMPOSE_FILE env var.

From https://docs.docker.com/compose/reference/envvars/#compose_file:

This variable supports multiple Compose files separated by a path separator (on Linux and macOS the path separator is :, on Windows it is ;). For example: COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml. The path separator can also be customized using COMPOSE_PATH_SEPARATOR.

[laugimethods]

@dermeister0 You could also permanently merge those files with tools like https://github.com/ImmobilienScout24/yamlreader :

> yamlreader common/common.yml prod/docker-compose.yml > docker-compose-prod.yml
> docker-compose -f docker-compose-prod.yml -p myproject up --build

> cat docker-compose-prod.yml
services:
    web:
        build: .
        environment:
            DOMAIN: null
            NEW_RELIC_KEY: null
            PREFIX: null
        image: the-prod-image:latest-release
        ports:
        - 80:80
        - 80:443

[teodorescuserban]

@dermeister0 thank you for your suggestion unfortunately it is a clunky way out of it. Using extends remove the need to actually know how exactly you need to daisy chain those. While I can live with doing that for myself, I could never apply this solution to my dear devs.

[teodorescuserban]

However, I was not aware that COMPOSE_FILE env variable can contain multiple values. Thank you @gsong ! That is awesome and can use it (I define it in the .env file). There is one single issue here: in the base / common files I might also have some services I dont need.

Take for example a flat common file defining database and web containers. On staging you want them all clumped together but on prod you would want separate hosts for db and for web.

[dermeister0]

Also, docker-compose.override.yml is loaded by default.

https://docs.docker.com/compose/extends/#understanding-multiple-compose-files

[dedalozzo]

I use this approach with version 3.3:

• put common configuration options and services in docker-compose.yml;
• use docker-compose.override.yml for specific development configuration and services (like xdebug for example);
• use docker-compose.staging.yml for specific staging configuration options.

Note: I don’t run Docker in production.

Using this approach I can easily build locally using docker-compose build and when I deploy on staging I use:

docker-compose -f docker-compose.staging.yml -f docker-compose.yml build

I’m using Apache, and I don’t have separate virtual host files for development and staging. I have spent quite a bit to avoid having different files. In the end I have seen the only valid approach is using <IfDefine> and environment variables (that I set in the environment section of the yml files), to include ssl configuration for example. And I use TLD and domain prefix, so I can have something like www.example.local http://www.example.local/:8080 and www.staging.example.com http://www.staging.example.com/. Locally the websites run under http and on staging on https. With this approach I don’t have to maintain different versions of the files. I think the same could be done in production, but as I said I prefer avoiding Docker in production, atm.

Paths are all relatives, so the containers will work in any environment.

My 2 cents

-Filippo

[Yajo]

In my case, previously I used something like this:

• common.yml
• devel.yml -> extending common.yml
• prod.yml -> extending common.yml
• docker-compose.yml -> local, git-ignored, symlink to the desired environment (devel or prod).

Thanks to https://github.com/moby/moby/issues/31101#issuecomment-329482917 and https://github.com/moby/moby/issues/31101#issuecomment-329512231, which I didn't know about, now I can move to v3 by using a different schema:

• docker-compose.yml -> what previously was common.yml
• devel.yml -> overriding docker-compose.yml
• prod.yml -> overriding docker-compose.yml
• docker-compose.override.yml -> local, git-ignored, symlink to the desired environment (devel or prod).

I can also do any needed hacks by using the env variable, so in my case the issue is fixed. Thanks! 🎉 (Sorry, I should have read the new docs properly before complaining).

PS: Still, extends would be a good thing to have back, but at least we have a fair enough alternative. 😊

[schmunk42]

@shin- First I was very disappointed to see the extends feature missing in 3.0, but YAML anchors (example: https://github.com/JanNash/docker-noop) would be a more than sufficient replacement.

The only thing is that, like in the above example, you need to put your anchors in some valid section of your docker-compose.yml file.

Could we get a (top-level) templates property or hidden keys like in GitLab to be more flexible in the way where to define the anchors?

[rdxmb]

@schmunk42 have a look at https://github.com/docker/cli/pull/452

[grayside]

The problem with relying docker-compose.override.yml is that it presupposes you only have one type of override you need to include, and one layer of overrides.

In my case, I want to have certain behaviors common to all local development, but might need them to be subtly different if the developer is running Windows, OSX, or Linux. To keep this manageable in a docker-compose v3 context, I have Linux users operating on the same behavior as the staging environment, which works but means they are slightly out of step with other developers.

I avoid using -f for local development because I've found it's very prone to human error, and relying on it as a toggle for too many things causes trouble. Selecting one file seems reasonable, selecting multiple is something I carefully avoid imposing.

[teodorescuserban]

FYI, I think that once the hack with x- mentioned above, and also daisy chaining docker-compose files using the COMPOSE_FILE variable in the .env project file, adding the docker-compose.override.yml should solve every use case I have encountered so far.

The yml anchors are also a nifty thing that I intend to use in the near future.

Not very happy with the x- hack beauty but I can live with that.

Thank you guys for your input!

[pnickolov]

It seems like the removal of extends is causing a lot of heartburn and either blocking move to v3 or resorting to hacks. I like the ability to define a "template" (or abstract) service that can carry some common things (e.g. update policies)

I am working on a simple golang filter utility that can pre-process a docker compose file with extends and spit out a clean v3 file with the extends resolved:
resolve-compose docker stack deploy -c docker-compose.yaml
(or docker-compose up)

Will this work for at least some of your use cases / why gotchas?

[ABitMoreDepth]

@pnickolov That would be gorgeous for me.

I've been looking ansible-container (I use ansible personally, but not yet at work), which looks like it would do everything I need, but a script like yours would be more preferable (less churn)

So long as it can recursively process extends: keys, I'd be happy!

[laugimethods]

Ok, looks like we found alternatives to extends... 👍

What's disturbing is the fact that Moby's Project Management doesn't seem to consider maintaining compatibility as a key element. Forward compatibility is essential for broader adoption, especially for large applications deployed into production. How could we push for Docker Swarm / Docker EE when there is no guarantee that core features like extends will remain? 👎
A good reputation is hard to win and easy to lose...

[schmunk42]

Personally, I'd prefer to rely on native YAML features to accomplish the task which is handled by extends in v2 syntax, rather than a custom-script or a larger solution like ansible. I was having similar problems earlier and started writing my own converter before there were solution like using multiple .yml files etc.. (https://github.com/schmunk42/yii2-yaml-converter-command) - but it wasn't a viable solution.

To me it's also fine to deprecate features, if this is done along with a versioning policy; you can't carry old stuff forever, even if this means some work on our side.

[laugimethods]

@schmunk42 Deprecating features is fine, but only when:

• the removed feature is no more really used or is a real blocker for evolutions
• it is announced in advance
• a migration path is provided right away

Unfortunately, none of those requirements (in my book) does apply to the deprecation of extends...

[schmunk42]

@laugimethods What's the reason you need to use v3?

[laugimethods]

@schmunk42 Because of Swarm:

Version 3.x, the latest and recommended version, designed to be cross-compatible between Compose and the Docker Engine’s swarm mode. This is specified with a version: '3' or version: '3.1', etc., entry at the root of the YAML.

The docker stack deploy command supports any Compose file of version “3.0” or above

Btw, my comment is not only related to extends, but to any (painful) deprecation that might occur in the future...

[schmunk42]

Yeah, we are facing the same issue because of swarm mode.

I asked myself in the first place, why the hell the docker CLI is now able to use --composer-file input. But from our learnings with docker/swarm it looks like a pretty complicated thing to run stacks on a (self-managing) swarm and that there are several good reasons why this have been moved to the engine.

[schmunk42]

Just to note some of my findings here ... transitioning from v2 to v3.4 is far from being easy.

Some of my problems:

• there are other unsupported options like volumes_from, rather easy to circumvent, since we wanted to remove that anyway
• .env files (like with docker-compose) have no effect with Docker CLI
• specifying multiple compose files (docker stack deploy -c docker-compose.yml -c docker-compose.override.yml doro) seem not to work correctly, there's no error, but it looks to me like they are also not merged correctly - but there's also no command like docker-compose config to check it
• there's no "easy to install" (pre-release) binary for docker-compose which supports v3.4 syntax; like Docker edge
• external networks need to be created with --scope swarm

CC: @handcode

[djeeg]

Using the latest builds

• docker 17.09.0-ce
• docker-compose 1.17.0dev

I am now using this configuration pipeline to replace my use of _extend_ and _noop_
Keeps things a little bit DRYer
Hope this helps someone

base.yml (shared definitions, this is a valid yaml document, so mergable using _docker-compose config_)

version: '3.4'
networks:
  net_back:
    external: true

base-inject.yml (shared anchor definitions, unfortunately cant be added to base.yml as anchors can not be referenced across different yaml files, instead these are injected as text into foo.yml, not an ideal way to do this)

x-logging: &logging
  driver: json-file
  options:
    max-size: "50m"
    max-file: "2"

foo.yml (generic stack definition, references objects from base.yml, anchors from base-inject.yml and objects overridden in foo-dev.yml)

version: '3.4'
[[base-inject]]
services:
  foo:
    image: ${DOCKER_REGISTRY}/foo:${IMAGE_VERSION}
    volumes:
      - type: volume
        source: "volfoo"
        target: "/foo"
    networks:
     - net_back
    logging:
      <<: *logging

foo-dev.yml (per environment stack definition)

version: '3.4'
services:
  foo:
    ports:
      - "8080:80"
volumes:
  volfoo:
    name: '{{index .Service.Labels "com.docker.stack.namespace"}}_volfoo_{{.Task.Slot}}'
    driver: local

Then the deployment command:

docker stack rm stack_foo && echo "waiting..." && sleep 3 &&
  cat foo.yml | sed -e '/base-inject/ {' -e 'r base-inject.yml' -e 'd' -e '}' > ./foo-temp1.yml &&
  export $(sed '/^#/d' ./dev.env | xargs) &&
  docker-compose -f base.yml -f ./foo-temp1.yml -f foo-dev.yml config > ./foo-temp2.yml
  docker stack deploy --with-registry-auth --prune --compose-file ./foo-temp2.yml stack_foo

1. Remove old stack
2. Wait for stack removable to propagate
3. Inject anchors into generic defintion, save to tmp file
4. Read+set env file variables from file
5. Use docker-compose to merge the three files together
6. Deploy merged file

[efrecon]

For those of you craving after extends in compose 3+ files, I have just put my hands on a tool called baclin. baclin linearises such directives, recursively replacing all extends directives by their content. This is alpha software, the code is part of my machinery as I am currently writing code to support Swarm mode and stacks deployment. Platform binaries of an early version of baclin are available here. Please report any comment or issue here.

[jottr]

This is really confusing!
Reading the doc for the latest docker release (v17.09) and also the v17.06 release doc this feature should be available.

$ head -n1 docker-compose.yml
version: '3'

But compose up yields

ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.admin_application: 'extends'

when using the extends keyword.
Also, I can't find anything about the removal of extends in the compose changelog.

Now which is it?!
Crucial info like this shouldn't be hard to find or hidden inside some obscure github issue.

---

$ docker --version
Docker version 17.09.0-ce, build afdb6d4

$ docker-compose --version
docker-compose version 1.16.1, build 6d1ac21

[lemats]

@jottr, See docs:

The extends keyword is supported in earlier Compose file formats up to Compose file version 2.1 (see extends in v1 and extends in v2), but is not supported in Compose version 3.x.

So if you want to use extends you need to stick up to version: '2.1'.

[jottr]

My bad. It should still be at the top of the doc with a red deprecation warning sign.

[rdxmb]

My bad. It should still be at the top of the doc with a red deprecation warning sign.

@jottr Just use Github to create a separate issue for that or even create a PR for that. Just created the issue hiere: https://github.com/docker/docker.github.io/issues/5340

[sandrocsimas]

In my case I have the docker-compose-override.yml as follow:
yaml version: "3.4" services: common: extra_hosts: - "host1:172.28.5.1" - "host2172.28.5.2" - "host3:172.28.5.3" networks: default: external: name: "common-network"
And I have several other docker-compose.yml files that need to share the network and extra_hosts. How to do this using without extends feature?

yaml version: "3.4" services: mongo: image: "mongo" container_name: "mongo" hostname: "mongo" volumes: - "/opt/docker/mongo/default.conf:/usr/local/etc/mongo/mongod.conf" - /opt/data/mongo:/data/db" ports: - "27017:27017" command: "mongod --config /usr/local/etc/mongo/mongod.conf" networks: default: ipv4_address: "172.28.5.4"
It would be great if docker-compose supported yaml anchor and references between different files. Maybe, applying anchor and references after merging files.
For example:
yaml version: "3.4" services: common: &common extra_hosts: ... networks: ...
yaml version: "3.4" services: mongo: <<: *common image: "mongo" container_name: "mongo" ...
The result should be:
yaml version: "3.4" services: mongo: image: "mongo" container_name: "mongo" extra_hosts: // EXTRA HOSTS HERE networks: ...

[rdxmb]

@sandro-csimas this is possible with: https://docs.docker.com/compose/compose-file/#extension-fields
@shin- could this ticket be closed?

[Puneeth-n]

@rdxmb I don't think so. From what I can see, you cannot extend from another docker-compose file

[rdxmb]

Am I right in thinking that this is an issue for https://github.com/docker/compose/issues ?

Some debugging:

# cat docker-compose.yml 
version: "3.4"
services:
  foo-not-bar:
    << : *common
  foo-bar:
    << : *common
    environment:
      - FOO=BAR 

x-common-definitions-for-all-our-services:
  &common
    image: phusion/baseimage
    environment:
      - FOO=NOTBARBYDEFAULT

This is working with
docker stack deploy -c docker-compose.yml test

When using docker-compose:

# docker-compose up 
ERROR: yaml.composer.ComposerError: found undefined alias 'common'
  in "./docker-compose.yml", line 4, column 10

Changing the yml-file to:

version: "3.4"

x-common-definitions-for-all-our-services:
  &common
    image: phusion/baseimage
    environment:
      - FOO=NOTBARBYDEFAULT

services:
  foo-not-bar:
    << : *common
  foo-bar:
    << : *common
    environment:
      - FOO=BAR

also works with docker-compose.

So thought this should also be working with multiple files, which is not the case:

# docker-compose -f compose-services.yml -f compose-default.yml config > docker-compose.yml
ERROR: yaml.composer.ComposerError: found undefined alias 'common'
  in "./compose-services.yml", line 5, column 10
t# docker-compose -f compose-default.yml -f compose-services.yml config > docker-compose.yml
ERROR: yaml.composer.ComposerError: found undefined alias 'common'
  in "./compose-services.yml", line 5, column 10

# cat compose-services.yml 
version: "3.4"

services:
  foo-not-bar:
    << : *common
  foo-bar:
    << : *common
    environment:
      - FOO=BAR 

# cat compose-default.yml 
x-common-definitions-for-all-our-services:
  &common
    image: phusion/baseimage
    environment:
      - FOO=NOTBARBYDEFAULT

However, of course, merging this is possible with a simple use of cat:

# cat compose-default.yml compose-services.yml > docker-compose.yml && docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating network "test_default" with the default driver
Creating test_foo-bar_1 ... 
Creating test_foo-not-bar_1 ... 
Creating test_foo-bar_1
Creating test_foo-bar_1 ... done

Running on xenial with the versions:

# docker --version
Docker version 17.11.0-ce, build 1caf76c
# docker-compose --version
docker-compose version 1.17.0, build ac53b73

[sandrocsimas]

@rdxmb, thanks!
So I should go merging compose files using "cat" command and executing the final file.

[krjensen]

I also use the docker-compose config to do this. An example for environment specific settings:

This is run by the CI pipeline: docker-compose -f docker-compose.yml -f docker-compose.override.prod.yml config > docker-compose.prod.yml and then I use docker stack deploy -c .\docker-compose.prod.yml my-stack

[yeasy]

Vote for this, the extends is very useful for v3.

[liyaka]

Used it a lot with v2, very useful indeed!
+1

[djones]

I would love to see extends support in v3. It would help DRY up my docker-compose.yml file a lot.

[hadsed]

It has been almost a year since this issue has been brought and it is really obvious that a ton of people need this feature. Yet I haven't read a Docker dev respond to this request at all, nor an explanation of why it was not included in docker-compose v3 before release.

Sad state of software if we cannot even communicate or maintain features for our customers who put faith in a new technology.

[aCandidMind]

One possible interpretation (and maybe also a summary of the current thread) of the situation is like this:

• YAML anchors/references plus 3.4's extension fields achieves almost the same.
• multi-file can be made to work. See the simple cat and the advanced approach in this thread. The simple approach comment shows an issue of docker-compose loading multiple files towards the end (has anybody created an issue for that?). If that was fixed in docker-compose you wouldn't even need the file merging at all. So to me people desiring multi-file support should continue in docker/compose/issues as @rdxmb proposed.
• Bringing back extends was considered (see GitHub project events, nice transparency here from the Docker team, thanks!) and you might interpret the outcome as "there's loads of other stuff that's strategically more important for them" but you can still write a pull request for extends I guess.

To me that's a perfectly understandable viewpoint.

[JanNash]

@aCandidMind Agreed.

[joaocc]

IMHO, even though the approaches mentioned by @aCandidMind work, they add complexity to a the simpler-and-cleaner mechanism provided by extends.
Maybe it's me, but moving a moderately complex extends configuration to extension fields becomes much harder to read and maintain.
After reading many comments and posts, it is still not clear to me why extends was dropped and what the advantages of this regression in capabilities are.

[tylerbuchea]

It is possible with a little bit of bash magic I put up a test repo so you can try it yourself.

Just structure your stack deploy command like this:

docker stack deploy --compose-file=<(docker-compose -f docker/prod.yml -f docker/dev.yml config) <stackname>

[dnmgns]

@tylerbuchea - the only downside to that bash magic is that you might get a WARNING: Some services (<service-name(s)>) use the '<key>' key, which will be ignored. Compose does not support '<key>' configuration. This can cause some confusion. But hey, it works 👍

[tylerbuchea]

@dnmgns you're right! Thanks for pointing that out. Like @joaocc said nothing is going to beat native support, but the solution I mentioned above is the best one I could find given that there are no dependencies other than bash.

[dnmgns]

@tylerbuchea One dirty way is to just redirect stderr to /dev/null :)
docker stack deploy --compose-file=<(docker-compose -f docker/prod.yml -f docker/dev.yml config 2> /dev/null) <stackname>

[tylerbuchea]

Ain't no shame in it 😄

[g-borgulya]

I think the documentation should be more explicit about this confusion around extend.
The description of extend redirects the user to the how-to-upgrade docs, and the how-to-upgrade docs point back to the description of extend for more information. This is not helpful enough: as a user I'd expect some help how to deal with this whole problem, what options I do have and what I should consider. I'm very sure there was a clear idea behind the decision of removing extend from v3.

See:
https://docs.docker.com/compose/extends/#extending-services
https://docs.docker.com/compose/compose-file/compose-versioning/#upgrading

[Lucas-C]

Regarding @tylerbuchea great one-liner bash based solution,
sadly it does not support some advanced Docker stack features:

WARNING: Some services (web) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
WARNING: Some services (web) use the 'configs' key, which will be ignored. Compose does not support 'configs' configuration - use `docker stack deploy` to deploy to a swarm.

[vdemeester]

Not that since https://github.com/docker/cli/pull/569 go merged, starting from 18.03, docker stack deploy will support merging multiple compose file into one. It does not fully replace the extends key from composefile format v2 but hopefully it covers way more use case 👼

[Lucas-C]

My own workaround was to use yq (can be combined into a one-liner if using Bash):

yq merge --overwrite docker-stack.yml docker-stack.preprod.yml > merged-docker-stack.yml
docker stack deploy -c merged-docker-stack.yml preprod

[tylerbuchea]

@Lucas-C they're just warnings the output will still include your deploy and config keys. You can verify this if you run docker-compose -f docker/prod.yml -f docker/dev.yml config

[ezh]

It is for compose files v3.4 and higher. It supports cross yaml references (partial). I finished with this zsh alias/perl script:

alias regen=$'perl -MFile::Slurp=read_file -MYAML=Load,Dump -MHash::Merge::Simple=merge -E \'
  local $YAML::QuoteNumericStrings = 1;
  $n=read_file("/data/docker-compose.yml");
  $s=Dump(merge(map{Load($n.read_file($_))}@ARGV));
  local $/ = undef;
  $s =~ s/\\bno\\b/"no"/g;
  say $s;
  \' $(find /data -mindepth 2 -maxdepth 4 -name docker-compose.yml) >! /data/x-docker-compose.yml'
regen
export COMPOSE_FILE=/data/x-docker-compose.yml

1. read /data/docker-compose.yml with common part.
2. find all docker compose recursively (For example there are about 40 different containers/docker-compose.yml files in this project)
3. prepend each docker-compose.yml with /data/docker-compose.yml content
4. merge
5. save result to /data/x-docker-compose.yml

Pros: perl is common tool, all perl modules too, generation is fast.
Cons: I hate hacks but there is no other way for DRY. Final docker-compose is about 900 lines. Do you really want that I support it as a single file from the beginning? It is a shame to have binary docker wrapped with python docker-compose wrapped with perl hack.

[bitfinity]

How can you just take out a feature like extends? That seems like a core feature.

[shaun-blake]

Using docker-compose config piped to the standard input option of docker stack deploy -c - has solved the problem for me:

docker-compose -f docker-compose.yml \
               -f docker-compose.extended.yml \
               config \
| docker stack deploy -c - my-stack

[shaun-blake]

I haven't tried this, but I also just noticed this in the docker stack deploy documentation:

If your configuration is split between multiple Compose files, e.g. a base configuration and environment-specific overrides, you can provide multiple --compose-file flags.

With this as the example:

docker stack deploy --compose-file docker-compose.yml -f docker-compose.prod.yml vossibility

https://docs.docker.com/engine/reference/commandline/stack_deploy/#compose-file

[cjerdonek]

Is the rationale for removing extends documented somewhere? It doesn't seem to be explained in the official docs, e.g. here: https://docs.docker.com/compose/extends/#extending-services
If users could understand the rationale, then users could develop a better idea of how to respond to the removal. Thanks.

[bitfinity]

@shaun-blake I ended up using the multiple compose files. That seems to be the approach people use. Rather than inheritance it's more like a mix-in. When building or running I copy the correct environment yaml template to docker-compose.override.yml.

[drzraf]

Multiple docker compose files (eg: base.yml, local.yml, prod.yml) does not let service use YAML anchors from other files so factored service definitions could not be defined among multiple yml files.
Note, this issue is the 13th most commented: https://github.com/moby/moby/issues?q=is%3Aissue+is%3Aopen+sort%3Acomments-desc and 3rd most liked.

[Billy-]

If users could understand the rationale, then users could develop a better idea of how to respond to the removal. Thanks.

+1 on documentation on rationale for removing extends in the first place...

[xhliu]

Still no _extends_ after almost 1 and a half year later. Cmon, devs, u don't remove sth without giving an alternative.

[dedalozzo]

They did, they offer an alternative called composition. Please read my answer in the thread.

-Filippo

On 30 Jul 2018, at 09:41, Xiaohui Liu [e-mail] wrote:

Still no extends after almost 1 and a half year later. Cmon, devs, u don't remove sth without giving an alternative.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/moby/moby/issues/31101#issuecomment-408790200, or mute the thread https://github.com/notifications/unsubscribe-auth/AAS_0AOynjpfVnVo4ZqciMbmjBmkcTQ3ks5uLsbNgaJpZM4MDcml.

[teodorescuserban]

@dedalozzo, "in the thread" == ?

[dedalozzo]

Please see my comment here:

https://github.com/moby/moby/issues/31101#issuecomment-329527600 https://github.com/moby/moby/issues/31101#issuecomment-329527600

Basically you have to use a chain of .yml files to override or change the configuration of your containers.

Please read "Specifying multiple Compose files”

You can supply multiple -f configuration files. When you supply multiple files, Compose combines them into a single configuration. Compose builds the configuration in the order you supply the files. Subsequent files override and add to their predecessors.

https://docs.docker.com/compose/reference/overview/ https://docs.docker.com/compose/reference/overview/

This approach uses composition over inheritance to obtain, more or less, the same result.

On 30 Jul 2018, at 15:23, Serban Teodorescu [e-mail] wrote:

@dedalozzo https://github.com/dedalozzo, "in the thread" == ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/moby/moby/issues/31101#issuecomment-408880809, or mute the thread https://github.com/notifications/unsubscribe-auth/AAS_0FZO30NplqHRid_Id8VBOJW7nk5Iks5uLxbigaJpZM4MDcml.

[luckydonald]

Wouldn't we be able to get the same extendability back if we combine
yaml extension fields (compose 2.1+/3.4+)
with allowing those x- fields to reference other files?

So we could allow a root include listing to specify files to load.
they would be put in an x-include and be instantly usable via standard YAML anchors & merging.

---

Current compose v2.1+

# /docker-compose.yml
version: '2.1'

volumes:
  nginx_file_sockets:
    external: false
    driver: local

services:
  reverse_proxy:
    extends:
      file: reverse_proxy/docker-compose.yml
      service: proxy
    restart: 'always'
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - nginx_file_sockets:/sockets/nginx

  reverse_proxy_test:
    extends:
      file: reverse_proxy/docker-compose.yml
      service: proxy
    restart: 'always'
    ports:
      - "8080:80"
      - "8443:443"
    volumes:
      - nginx_file_sockets:/sockets/nginx

  web:
    extends:
      file: webservice/docker-compose.yml
      service: app
    restart: 'always'
    environment:
      ENVIRONMENT: 'production'
      DB_USER: ${WEB1_DB_USER}
      DB_PASSWORD: ${WEB1_DB_PASS}
    volumes:
      - nginx_file_sockets:/sockets/nginx

  web_staging:
    extends:
      file: webservice/docker-compose.yml
      service: app
    restart: 'no'
    environment:
      ENVIRONMENT: 'staging'
      DB_USER: ${WEB1_DB_USER}
      DB_PASSWORD: ${WEB1_DB_PASS}
    volumes:
      - nginx_file_sockets:/sockets/nginx

# /proxy/docker-compose.yml
version: '2.1'
services:
  proxy:
    build: ./
    volumes:
      - /certs:/certs:ro

# /webservice/docker-compose.yml
version: '2.1'
services:
  app:
    build:
      context: ./folder
      args:
        LINUX_VERSION: 20.s
        LINUX_FLAVOR: dash
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - bootstrap.memory_lock=true
    ulimits:
      memlock:
        soft: -1
        hard: -1

---

Idea of compose v3.X

# /proxy/docker-compose.yml
version: '3.9'
services:
  proxy:
    &proxy
    build: ./
    volumes:
      - /certs:/certs:ro

# /webservice/docker-compose.yml
version: '3.9'
services:
  app:
    &app
    build:
      context: ./folder
      args:
        LINUX_VERSION: 20.s
        LINUX_FLAVOR: dash
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - bootstrap.memory_lock=true
    ulimits:
      memlock:
        soft: -1
        hard: -1

# /docker-compose.yml
version: '3.9'
include:
  - /proxy/docker-compose.yml
  - /webservice/docker-compose.yml

volumes:
  nginx_file_sockets:
    external: false
    driver: local

services:
  reverse_proxy:
    << : *proxy
    restart: 'always'
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - nginx_file_sockets:/sockets/nginx

  reverse_proxy_test:
    restart: 'always'
    << : *proxy
    ports:
      - "8080:80"
      - "8443:443"
    volumes:
      - nginx_file_sockets:/sockets/nginx

  web:
    << : *app
    restart: 'always'
    environment:
      ENVIRONMENT: 'production'
      DB_USER: ${WEB1_DB_USER}
      DB_PASSWORD: ${WEB1_DB_PASS}
    volumes:
      - nginx_file_sockets:/sockets/nginx

  web_staging:
    restart: 'no'
    extends:
      file: web1/docker-compose.yml
      service: app
    environment:
      ENVIRONMENT: 'staging'
      DB_USER: ${WEB1_DB_USER}
      DB_PASSWORD: ${WEB1_DB_PASS}
    volumes:
      - nginx_file_sockets:/sockets/nginx

---

Diff

@@ /proxy/docker-compose.yml @@
-version: '2.1'
+version: '3.9'
 services:
   proxy:
+    &proxy
     build: ./
     volumes:
       - /certs:/certs:ro
 ```

 ```diff
 @@ /webservice/docker-compose.yml @@
-version: '2.1'
+version: '3.9'
 services:
   app:
+    &app
     build:
       context: ./folder
       args:
         LINUX_VERSION: 20.s
         LINUX_FLAVOR: dash
     environment:
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
       - bootstrap.memory_lock=true
     ulimits:
       memlock:
         soft: -1
         hard: -1
 ```

 ```diff
 @@ /docker-compose.yml @@
-version: '2.1'
+version: '3.9'
+include:
+  - /proxy/docker-compose.yml
+  - /webservice/docker-compose.yml

 volumes:
   nginx_file_sockets:
     external: false
     driver: local

 services:
   reverse_proxy:
-    extends:
-      file: reverse_proxy/docker-compose.yml
-      service: proxy
+    << : *proxy
     restart: 'always'
     ports:
       - "80:80"
       - "443:443"
     volumes:
       - nginx_file_sockets:/sockets/nginx

   reverse_proxy_test:
-    extends:
-      file: reverse_proxy/docker-compose.yml
-      service: proxy
+    << : *proxy
     restart: 'no'
     ports:
       - "8080:80"
       - "8443:443"
     volumes:
       - nginx_file_sockets:/sockets/nginx

   web:
-    extends:
-      file: webservice/docker-compose.yml
-      service: app
+    << : *app
     restart: 'always'
     environment:
       ENVIRONMENT: 'production'
       DB_USER: ${WEB1_DB_USER}
       DB_PASSWORD: ${WEB1_DB_PASS}
     volumes:
       - nginx_file_sockets:/sockets/nginx

   web_staging:
-    extends:
-      file: webservice/docker-compose.yml
-      service: app
+    << : *app
     restart: 'no'
     environment:
       ENVIRONMENT: 'staging'
       DB_USER: ${WEB1_DB_USER}
       DB_PASSWORD: ${WEB1_DB_PASS}
     volumes:
       - nginx_file_sockets:/sockets/nginx
 ```
<hr>
Resulting in the final version, which should be already yaml parsable:

```yml
# /docker-compose.yml
version: '3.9'
#include:
#  - /proxy/docker-compose.yml
#  - /webservice/docker-compose.yml
x-include:
  /proxy/docker-compose.yml:
    version: '3.9'
    services:
      proxy:
        &proxy
        build: ./
        volumes:
          - /certs:/certs:ro
  /webservice/docker-compose.yml:
    version: '3.9'
    services:
      app:
        &app
        build:
          context: ./folder
          args:
            LINUX_VERSION: 20.s
            LINUX_FLAVOR: dash
        environment:
          - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
          - bootstrap.memory_lock=true
        ulimits:
          memlock:
            soft: -1
            hard: -1

volumes:
  nginx_file_sockets:
    external: false
    driver: local

services:
  reverse_proxy:
    << : *proxy
    restart: 'always'
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - nginx_file_sockets:/sockets/nginx

  reverse_proxy_test:
    << : *proxy
    restart: 'no'
    ports:
      - "8080:80"
      - "8443:443"
    volumes:
      - nginx_file_sockets:/sockets/nginx

  web:
    << : *app
    restart: 'always'
    environment:
      ENVIRONMENT: 'production'
      DB_USER: ${WEB1_DB_USER}
      DB_PASSWORD: ${WEB1_DB_PASS}
    volumes:
      - nginx_file_sockets:/sockets/nginx

  web_staging:
    << : *app
    restart: 'no'
    environment:
      ENVIRONMENT: 'staging'
      DB_USER: ${WEB1_DB_USER}
      DB_PASSWORD: ${WEB1_DB_PASS}
    volumes:
      - nginx_file_sockets:/sockets/nginx

[Yajo]

AFAIK that's a YAML feature, built into the laguage spec itself, to avoid repeating parts in the same file. When using different files, that's basically impossible.

You should propose that feature to the YAML spec itself.

[blaggacao]

This discussion burns down to:

• How docker-compose -f file.yml is so much better than docker-compose -f file.yml -f file_extension.yml ?
• Or: wiring at the command level _vs_ wiring at the file level.

Only when working on the command line, the inconvenience becomes notable. We have to acknowledge that for a second. Everything else is scriptable, anyway.

If that's the real argument, then docker-compose up service has better semantics than docker-compose -f service.yml up: define everything needed for local development (aka on the command line) in docker-compose.override.yml.

Given semantics are very clean and well thought through. Adopting service.yml _for the command line use_ probably means lowering UX. There is another argument to it: while it is clear at first sight, what a docker-compose.yml is, a service.yml can be anything, really anything.

Disclaimer: A provocative opinion. :wink: I didn't take into account every possible use case...

[soubhik-c]

After this prolonged discussion not sure whether it will ever make it. IMHO, extends was cool and must have been atleast carefully deprecated and then discussed instead of simply dropping it away.

[BretFisher]

I think one thing we could do is improve the documentation story about v2 vs. v3. Many assume that v3 replaced v2, but that's not entirely true. Both receive new features that focus on their use cases. This GH issue was started so we could have the discussion about what future features needed to make their way from docker-compose into Swarm, and how to improve docs for using docker-compose, the compose file format, and Swarm stacks together. Extends still works great in the up-to-date v2.4. Hopefully, I can help offer information on what solutions we have today:

v2: For docker-compose cli only. Development workflow focused on a single machine and engine. Also good for CI build/test workflows. This version branch received new features as recent as December 2017 in v17.12

v3: Ideal for Swarm/Kube stacks, with multi-node concepts and maintains support for most docker-compose cli features.

If you're not using Swarm or Docker Enterprise Kubernetes stacks, there is no reason to use v3. Stick with v2.4, and you get all the docker-compose cli features including extends, depends_on, extension fields, and even depends_on with healthchecks (to avoid wait-for-it scripts).

v3 was created to try and merge the features of a single engine docker-compose cli world with a multi-node cluster world. Not all v2 features (like depends_on) make sense in a cluster. Other features (like extend) just haven't made it into v3, likely because before v3 existed, all the code was in docker-compose Python, and for v3.0 to support Swarm, they had to rewrite that in docker cli Go, and now they are writing it again in the engine daemon to eventually make a Swarm stacks API, which doesn't yet exist.

For those new to this issue, also note much of the work that's gone on to solve various configuration, templating, and team workflow concerns since 3.0 release:

• 3 features I briefly discuss in this article (env vars, templating, and compose scope)
• override files, docker-compose config, and .env files
• docker-app experimental templating and packaging cli. CHECK THIS OUT
• forthcoming docker-compose GUI (teaser at DockerCon 2018 keynote)

[krp]

The docs at https://docs.docker.com/compose/extends/#extending-services should emphasize in red the fact that the extends keyword is removed in v3, as it's more important than just a _note_.
I migrated and skimmed the docs for why it was no longer working, then followed several closed issues before ending up here, then went back to the original docs and noticed the wording.

The extends keyword is supported in earlier Compose file formats up to Compose file version 2.1 (see extends in v1 and extends in v2), but is not supported in Compose version 3.x.

Could be rephrased as:

The extends keyword has been removed in Compose version 3.x, but is still supported in earlier Compose file formats up to Compose file version 2.1 (see extends in v1 and extends in v2).

It's a small difference, but one that's easy to overlook when skimming docs.

[BretFisher]

@krisrp PR started ^^^

[krp]

Thanks @BretFisher

Are there any plans to perhaps rename v2 to "version: docker-cli" and v3 to "version: swarm/kube"?
It'd make more sense to differentiate them like this, considering how v3 supersedes v2 in most other versioning schemes. At present both are maintained and diverging, so unless I'm mistaken it seems like they'll both be around for a while.

[cpuguy83]

@krisrp On the contrary, the reason for incrementing a major version number is to signal the divergence in compatibility.

[krp]

@cpuguy83 I wasn't implying otherwise. Apologies for not being more clear or explicit.
IIRC Gnome 2 & 3 also had this confusion years ago when independent forks of each were maintained.

I don't want to derail this thread to discuss the semantics of versioning (bad pun), so I'll leave it at that. @BretFisher 's post last week about improving documentation on v2 vs v3 would have helped myself and probably others.

[luckydonald]

@shin- @cpuguy83 Looking at this issue over a year later, what _is_ the reasoning for not adding it back in version 3?

I couldn't find much about it, other than "we could do it differently" (but no actual better solution offered)
Is there a technical limitation? Or just a lack of pull request?

After all, my compose 2.1 files are still running fine.

[skorokithakis]

Not only this, but the changelog says "this has been removed, see 'how to upgrade' for details". I look at "how to upgrade" for, you know, details on how to upgrade, and what that says is "see 'extending services' for details". I go to "extending services" figuring I'll finally see a way to extend my files, only to see "this has been removed, see the changelog for details".

At this point, this seems like a cruel joke the documentation writer is playing.

[cpuguy83]

Ultimately, the "stack" format is not controlled here and is part of the Docker CLI.

I personally don't know the reason it was excluded from v3... I also don't think I've seen anyone actually try to add it in.
It might be best to bring this up in docker/cli... maybe even just a PR with a high level doc change that acts as if the feature is there so it can be discussed and the implementation can be added once the design is approved.

Basically, if someone wants it, make a PR. I suggest the doc change just to make sure you don't waste a ton of time in case it is rejected... since again I am not sure why it was omitted from v3.

[andrewjaanuu]

Same as above. Waiting for this to be solved before using docker-compose again.

[aminancelot]

+1 please get this fixed, we have extend based compose files and can't use compose for swarm because of it.

[M0NsTeRRR]

+1 for extends feature

[tabulon]

Any news on this?

[daniele-orlando]

Still waiting for it

[thomasrutkowski]

Same here. Still waiting.

[jaykishan007]

Any update?

[SC7639]

Was there ever a reason given as to why it was taken out?

On Mon, 5 Aug 2019, 11:10 Jaykishan, [e-mail] wrote:

Any update?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/31101?email_source=notifications&email_token=ABOE6GA4CXY6ESMZMTDSFGDQC74CZA5CNFSM4DANZGS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3RLFLQ#issuecomment-518173358,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOE6GCEFFJ3SOLDWRWX2IDQC74CZANCNFSM4DANZGSQ
.

[dev-ls]

Any update?

[vitalik]

so.. it's almost 3 years...
but I still have hope it will land :D

[EverybodyKurts]

There needs to be some sort of alternative to extends if it's not coming back. Why not allow abstract services? The file format would be flat and all service declarations would be in one file. You could use abstract services in conjunction with yaml's ability to add aliases for a node (via an &) reuse those aliases via the <<: operator.

[erodrig]

Why 3 years!! it seem you are working and focusing in stuff no one cares about

[gioppoluca]

any update on this?

[Cristian-Malinescu]

This is pathetic. Terraform uses composition - so it is doable, why compose is not capable to follow on this good design practice ?!
Terraform Modules Composition
Terraform Best Practices

[luckydonald]

"pathetic", nice.

@Cristian-Malinescu Go ahead, implement it please.
This is free and open source software.

Instead of complaining like every other one in this chat.
That just doesn't add any value here, really.
As said several times nobody did want to implement this so far,
so it's just a problem someone else should fix k,thanks.

[aminancelot]

@luckydonald Thanks to push away @Cristian-Malinescu with an easy/passive-agressive answer, it's, like always not helping. @Cristian-Malinescu It is do-able as it already been done before but removed, there must (I hope) be a reason.Is there anyone on this thread actually in the docker-compose team so he/she can shed light on the matter?

[nomasprime]

Skimmed the thread and leveraging supported YAML functionality has been mentioned.

Thought this example might help.

[arseniybanayev]

@nomasprime thank you for that find! I was deciding between v2 and v3 for my project, and this resolved the big dilemma in this thread. Surprised that these alternative solutions are not mentioned in the official docs for the extending services feature.

[mbentley]

Nice, sounds like a good opportunity to use the Request docs changes link on the right nav bar of the docs page.

[luckydonald]

@nomasprime Yep, that idea was around in this thread before.

If that could combined with a file loading mechanism for other yml files, that's all what's needed really to have all the old depends functionality.

See above, e.g. https://github.com/moby/moby/issues/31101#issuecomment-413323610

It wouldn't be very _readable_, but it would be at least _possible_.

[lucasbasquerotto]

@nomasprime thank you for that find! I was deciding between v2 and v3 for my project, and this resolved the big dilemma in this thread.

@arseniybanayev The article on medium says about v3 only, but newer versions of v2 also support anchors and extension fields. In my case I choose v2 (2.4 more specifically) because I use docker-compose and not swarm (and v3 doesn't support some features of v2 like limiting container memory)

[thaJeztah]

and v3 doesn't support some features of v2 like limiting container memory

v3 does support limiting memory, but the field is under deploy -> resources -> limits https://docs.docker.com/compose/compose-file/#resources

[lucasbasquerotto]

@thaJeztah I mean, for docker-compose (because I'm not using swarm in the project I was referring to in my previous comment). IIRC deploy is only for swarm, isn't it?

[gregorskii]

Would it make sense to make a separate config for swarm and local? Seems like these two are conflicting with each other. Understandably Docker would like swarm usage to be increased, but a lot of people only use compose for local development.

I for one have never used swarm and run containers in production with either ECS, k8s, or GAE.

[thaJeztah]

Most options should be translatable / usable for both swarm/kubernetes services and for containers deployed through compose. I'd have to check why memory limits would not be applicable for docker-compose

[yvess]

still missing the extends feature, but for my main use case I switched to multiple docker compose files via the COMPOSE_FILE env. I use it mostly to use the same base docker-compose.yml for dev and prod with different passwords or configuration.

example:

• on dev: export COMPOSE_FILE=docker-compose.yml` # default
• on prod: export COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml` # uses both yaml files

In docker-compose.prod.yml I just overwrite the env vars with the prod passwords.

This setup is simple and I don't need to always add multiple "-f" to the docker-compose command. I just need to set the COMPOSE_FILE env var different on the dev computer and the server and git ignore the docker-compose.prod.yml file.

[akateliev]

Still waiting :)

[dave-dm]

I've been using this as a way to extend:

docker-compose \
  -f ./docker/base.yml \
  -f ./docker/extended.yml \
  up

But extend in the file would be nicer, no need for an extra bash script.

I've also been using this to extend dynamically from the bash script:

extended_docker_compose="
  version: '3.5'
  services:
    my-service:
      restart: always
"

echo "$extended_docker_compose" | docker-compose \
  -f ./docker/base.yml \
  -f /dev/stdin \
  up

[FernandoMiguel]

@dave-dm those are plain old overrides!

[thatnerdjosh]

I believe this is a potential valid usecase for extends

https://github.com/NerdsvilleCEO/devtools/blob/master/doctl/docker-compose.yml#L10

I have a docker-compose wrapper which I use for an env of services https://github.com/nowakowskir/docker-compose-wrapper

EDIT: Another possible usecase is someone has a LAMP/LEMP stack and they want to extend those services for use with a specific service such as wordpress

[andrewjaanuu]

Still waiting since 2017 while exploring docker compose alternatives.

[nomasprime]

@nomasprime Yep, that idea was around in this thread before.

If that could combined with a file loading mechanism for other yml files, that's all what's needed really to have all the old depends functionality.

See above, e.g. #31101 (comment)

It wouldn't be very _readable_, but it would be at least _possible_.

@luckydonald thanks for pointing out. Odd there's no built-in YAML include functionality, would certainly solve a lot of issues.

Looks like it would be pretty easy to implement a third party solution though so not sure why this hasn't been brought over to v3 🤷‍♂

[MarechJ]

A little reminder that a lot of people would like this feature :)

What is the reason for not porting it to v3 btw?

[SC7639]

I forget but is there an actual reason it was taken away?

On Wed, 6 May 2020, 23:14 Julien Marechal, [e-mail] wrote:

A little reminder that a lot of people would like this feature :)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/31101#issuecomment-624919070, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOE6GGDIVGATP734YJA4UTRQHOLJANCNFSM4DANZGSQ
.

[a-abella]

The way yaml anchors handle this use case is a bit inconvenient compared toextend. It's power, in my opinion, came largely from recursive merging of elements. Anchors get you maybe 75% of the way there--they don't merge yaml recursively; all your merging happens at the top level. Referencing an anchored service template and then redefining the environment block will result in overwriting the anchored service's environment block instead of merging. You'll need to anchor and reference every dictionary down a recursive tree of dictionaries in order to match the behavior of the extend keyword.

An example:

# anchors for the service, environment, deploy, and deploy.placement blocks
# you'll need an anchor for every dict that you want to merge into
x-common-app: &common-app
  image: app:1.0
  environment: &common-app-environment
    common_option: a
    overwrite_option: b
  deploy: &common-app-deploy
    max-replicas-per-host: 3
    placement: &common-app-deploy-placement
      constraints:
        - 'node.labels.app_host==true'

services:
  myapp: << *common-app
    environment: << *common-app-environment
      foo: bar
      baz: xyzzy
      overwrite_option: quz
    deploy: << *common-app-deploy
      replicas: 15
      placement: << *common-app-deploy-placement
        preferences:
          - spread: node.labels.region

# The above yields the following:
services:
  myapp:
    image: app:1.0
    environment:
      common_option: a
      overwrite_option: quz
      foo: bar
      baz: xyzzy
    deploy:
      replicas: 15
      max-replicas-per-host: 3
      placement:
        constraints:
          - 'node.labels.app_host==true'
        preferences:
          - spread: node.labels.region

It might not look so annoying in this example, but it gets annoying and less readable if you're templating multiple (10+) services off one extension block.

In my mind an ideal scenario is a combination of the yaml anchor approach and the extend approach--allow extending only from a top level x- prefixed extension field block, with the smarter merging characteristics.

In my organization we found yaml anchors a bit syntactically sloppy so we basically re-implemented the extend feature in an external python script. That works for us, but it's a lame way to have to deal with something. Similarly we've had to create our own external tooling to deal with the removal of depends_on for v3/Swarm stacks.

[efrecon]

I have done a lot of gitlab CI YAML lately. It has exactly these features, which are sooo nice to achieve readable and manageable templates and final configurations:

• You can include other YAML files (good for templates)
• You can extend (even across projects/using remote resources through https). The extend documentation describe exactly what @a-abella describes for the compose format.
• You can also "hide" from being considered the real stuff. In compose format it's x-, in gitlab CI it's an initial . instead.

This is the exact set of feature that makes these files bearable.

[stellarpower]

I arrived at this issue from the docker docs, in my case I wanted to template my docker-compose setup, and as a 'workaround' I followed the advice above and did decide to look for an existing templating program. I won;t get those our hours back so am detailing a bit of what I found here, irrespective of any discussion on this actual feature request, however, it may be that those involved would feel the use of a YAML-based template system to generate a compose file rather than integrating this feature into docker-compose itself might be more appropriate in terms of encapsulation and picking the right tool for the job.

For some context, I am using a basic reverse proxy with Let's Encrypt and several application containers (Nextcloud for now, one for me, and some separated ones for friends) - in this case, I wanted to make a template of the Nextcloud containers so that I avoid mistakes and duplication with keystrokes for very similar setups. The following packages were what I tried:

• ytt
• Dhall
• CUE
• Jsonnet

ytt seems very comprehensive and is the only option to use YAML natively. It seemed powerful and the right tool for the job, and it uses Starlark, a superset of Python, directly inside the YAML file to performing processing. However after not long, the template became very messy, and the littering of fragments of code and fragments of YAML, plus mixing of Python data types like dictionaries and arrays and YAML fragments (that seem to be somewhat processed like text, a bit like using an HTML template engine that outputs tags like strings) eventually resulted in too many errors and too messy a file. Dhall also seems very comprehensive and uses a unique native language that can be output to a variety of formats; It seems more like a metaprogramming language than a template system, however given that the syntax is functional and it is pretty strictly typed, it quickly became more complex than was worth it for a simple template for unstructured YAML. As what seems a little like a blend of JSON and Haskell, it required too much thought to work what I needed done into the language.

Interestingly, sometihng that was difficult with both Dhall and ytt was using parameterised field names, both would have worked well enough otherwise, but I need to have the name of my instance appear in the services names and volume names, and in both of these achieving this was somewhat ugly; using arguments for the values in a hash is easy, but using those arguments in the names of the keys was messy or I couldn't find how to do it neatly, plus Dhall enforces type safety and this simply goes against that concept. With Jsonnet it was a simple as putting the expression in square brackets.

CUE and Jsonnet are both JSON-oriented, however running these through a converter is not hard at all, and it seems again, like Dhall, CUE has a lot of powerful features and was born out of shortcomings in Jsonnet, however partway into the documentation, it became apparent it was already overkill; perhaps with much more time to learn it properly it would be the superior option, but it seems CUE is more oriented to validation and schemata than a simple template job and so I quickly moved onto Jsonnet and finished the job quite quickly.

Finally, it was only after I had completed all this I realised the whole time I had been comparing these tools to the simplicity of Liquid tags or similar HTML templates, that I could actually probably simply have used Liquid in the first place. I have only used it in the context of a Jekyll site, so it just never ocurred ot me ot obtain a standalone package, however with basic looping and lists, and the ability ot evalate expressions straight into in-place text, this probably would have been a much better too for the job; Jsonify probably superior for JSON, but Liquid could operate in pure YAML and so the file becomes more readable again.

[jimwrightcz]

+1 docker-compose was one inspiration behind the bespoke solution I've implemented at work since this ticket was created to support migrating a large number of test envs to k8s. I was very careful to avoid bells and whistles but pretty quickly justified an analogous feature. The philosophical discussion (composition versus inheritance etc.) looks to me like a distraction from common sense (with the benefit hindsight - still unresolved nearly 3 years later). Evidently it is required by people who might continue to use docker-compose.

[darkn3rd]

+1 :+1:

I used this feature heavily before for dev/test/ci environments, where I could extend from a compose file in subdirectory paths of ./config/{dev,test,ci}/compose.yaml. I would have a .env that had COMPOSE_ENV=dev, but developers could override, and obviously I would override in ci.

I'm shocked by deprecating the feature and not replacing it with something that can do something similar. Maybe just allow us to use jinja2 and do what we want. I hope Docker-Compose would be less anti-DRY. :'(

[Vanav]

Seems that extends is supported by docker-compose since v1.27 (https://github.com/docker/compose/pull/7588).

[yvess]

One use case where I use the feature heavily, is to version docker images to code. My docker dev and prod compose files both extend from docker-images.yml where only the basic service is listed and a tagged version of the image of the service.

Didn't found an easy workaround for this in v3.