Compose: extends not supported in version 3 schema

[chaosk]

I'm running Docker 1.13.0-rc4 and current master (545153f117a0be523eeaaed05b2458700a9e856d) of Compose.

I tried to deploy a stack using following docker-compose.yml file:

version: "3"
services:
  base:
    image: hello-world
  extending:
    extends: base
    labels:
      - "label=test"

Got following in response:

$ docker stack deploy --compose-file ./docker-compose.yml test
Compose file contains unsupported options:

extends: Support for `extends` is not implemented yet. Use `docker-compose config` to generate a configuration with all `extends` options resolved, and deploy from that.

I found an issue about it in Docker's repo (docker/docker#29304) and attempted to run the command given by the error message, which resulted in:

$ docker-compose config
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.extending: 'extends'

[shin-]

cc @dnephin

[chaosk]

I've been wondering, wouldn't adding compose/config/config_schema_v2.1.json#L124-L140 to config_schema_v3.0.json work? I tried doing that locally and it generated output correctly, but I guess it could be breaking other uses.

[lemats]

Note for others struggeling with this issue:

Regarding to the docs:
"extend" is NOT yet supported with a version 3 compose file.

https://docs.docker.com/compose/compose-file/#extends

This option is not yet supported when deploying a stack in swarm mode with a (version 3) Compose file. Use docker-compose config to generate a configuration with all extends options resolved, and deploy from that.

[viztastic]

Do we know which version of docker 'extends' will be supported in compose v3? Even if just Indicative?

[byrnedo]

I've tried the docker-compose config command and it says the same error.
Resorted to perl: https://gist.github.com/byrnedo/7ac5e461ba0866299fdd94899bebdbd9

[nicodmf]

+1 for supporting extend in v3

[ABitMoreDepth]

Would it be a lot of effort to support the use of extends when using docker-compose not in swarm mode?

I use extends to manage production, development and test setups and would like to continue doing so outside of swarm mode, but eventually use docker stack deploy with the production variant. I can probably use @byrnedo's script above to achieve the latter, but I'm blocked running docker-compose up on a v3 file containing extends... :/

[nicodmf]

This is the same for me, extends is a really usefull option, which for know cause a hole in hour workflow deployment

[willseward]

The docs seem a bit misleading ATM:

Note: This option is not yet supported when deploying a stack in swarm mode with a (version 3) Compose file. Use docker-compose config to generate a configuration with all extends options resolved, and deploy from that.

It reads like docker-compose config can be used to generate a v3 docker stack deploy-compatible docker-compose.yml file, but that is obviously not the case.

[artheus]

@willseward I strongly agree with you on this. docker-compose config should obviously be able to use the extends configuration keyword even when using version 3, according to the documentation.

It's a very ugly hacky solution to do what I'm forced to do right now:

$ docker-compose config | sed "s#version: '2\.1'#version: '3'#g"

EDIT: This is especially ugly since docker-compose will treat it as 2.1 version syntax, so it will certainly force developers into hacky solutions which will probably also be limited on certain version 3 features.
Maybe some people will create their own tools to, for the time being, replace docker-compose. I sure am tempted to do that.

[shouze]

@dnephin @shin- @vdemeester can I contribute to add back support of extends for v3.x?

What have to be done? I've quickly looked & I guess that this project has to be patched, but at least https://github.com/aanand/compose-file too.

[shouze]

@dnephin ok so I've seen you've moved aanand/compose-file to https://github.com/docker/docker/commit/f5af9b9738892b5988f987ce5fbce6e31a10e768 so docker-compose & docker repos needs to be patched in fact.

[arun-gupta]

What was the reason for removing extends from v3?

[shouze]

@arun-gupta I guess it's because a lot of docker-compose files parsing is also now done in go in docker engine source code and it's not have been ported from python ATM cc @dnephin right?

If yes, the question is to know if it's planned to bring it back or no. Would be nice to also know - if it's a deprecation - if another mecanism is planned to merge some properties from some kind of abstract services (which is the main use case of extends for everyone I guess).

[nshCore]

found this issue, for the same reason as others have already mentioned.

I've upgraded to docker-compose v3 to take advantage of some new docker-swarm features, but now extends is no longer supported? WHY!!

It is such a useful option for writing clean dockerfiles and managing services in different environments.

+1 for bringing this back

[Richard-Mathie]

@arun-gupta welcome to how the docker dev's roll out new features, depreciating old sensible functionality for new stuff which hasn't been released yet. Its like a great wet fish in the face, This morning I was all like

yay finally compose 3 adds deploy so I can use it with swarm instead of loads of bash scripting.

20 seconds later

what they removed "extents"....whyyyyyy... there is no god!

[Electron-libre]

Damned I just spent the day to build a nice extend layout for our services.
Hopefully I do not need the v3 features and will be able to downgrade to the 2.1.

I feel less confident now to justify the time invested in our docker stuff :(
I hope it wont get worse

[tmornini]

This document is also rather misleading:

https://docs.docker.com/compose/compose-file/compose-versioning/

In the section "Upgrading Version 2.x to 3.x" there is no mention of extends being removed. :-(

[Electron-libre]

I am working on documentation fix.

[willseward]

I really believe that with the swarm inter-compatibility, the extends key becomes more important than ever

[shin-]

At the end of the day, this is dependent on Docker Engine supporting the feature -as a result, I created docker/docker#31101 to track that possibility. Please voice your concerns there if this feature is important to your project!

[willseward]

Thanks @shin-. I would even be happy if docker-compose config would flatten the files. Then I could use the resulting compose file with Swarm.

Maybe something like this?

docker-compose config -f docker-compose.yml -f docker-compose.prd.yml | docker stack deploy --

That'd be acceptable for the interim...

[shouze]

@willseward yes should be cool & a great 1st step probably... but couldn't address a last issue (which was a very kind feature): abstract services.

I don't know if you have some but personally - and I'm not the only one - I have this kind of configs very often in some common-services.yml files (that I use like that docker-compose config -f common-services.yml -f docker-compose.yml)

services:
    some_service-abstract:
        build:
            context: .
        environment:
            - SOME_VAR
        image: vendor/service:tag
        networks:
            - somenet
        logging:
            driver: gelf
            options:
                gelf-address: "udp://127.0.0.1:17110"
                env: SOME_VAR

    some_service:
        extends:
            file: ./common-services.yml
            service: some_service-abstract
        environment:
            - LISTEN_ON_PORT=8888
        expose:
            - "8888"
        command: server
        networks:
            - some_other_net

    some_service-worker:
        extends:
            file: ./common-services.yml
            service: some_service-abstract
        command: worker_cmd
        networks:
            - some_other_net_take_two

So basically @shin- @vdemeester @dnephin what are recommandations for this use case?
Do you recommend to duplicate that in every service & every service env? Is there any other tricks?

        build:
            context: .
        environment:
            - SOME_VAR
        image: vendor/service:tag
        networks:
            - somenet
        logging:
            driver: gelf
            options:
                gelf-address: "udp://127.0.0.1:17110"
                env: SOME_VAR

[shouze]

Also, as I'm convinced that composition has always been greater than inheritance... and as extends wasn't very clear about doing either composition or inheritance or both... I guess that to push the discussion forward. The BC break is here so we can see it as an opportunity to make a new & clear proposal.

To me the ideal thing would be:

• (Limited) Inheritance: To add a new abstract key at the root of the docker-compose file schema. This key can be use to describe abstract services. This way we either can reintroduce the extends key in 3.x schema but limited to abstract services. We can also avoid to use the same keyword to avoid confusion by naming it extends_abstract or whatever.
• Composition: spread the word that composition of services should be done whith multiple docke-compose files (the multiple -f option of docker-compose).

[nshCore]

@willseward solution is a great one, this not only solves the issue with no longer supporting extend, but it also solves the fact, you can't run docker-compose bundle on multiple docker files to deploy a docker stack.

Of course it would be alot easier for docker engine swarm mode to have better inter-compatibility with docker-compose to be able to deploy a stack spread across multiple compose files rather than having to implement a new feature to make a limited experimental feature work.

[Electron-libre]

I agree that composition lead to more modularity than inheritance.
But Inheritance as kind of hierarchy backed in. And parent definition can be override or completed by child one. This was the case with extends.

Composition on the other hand does not include hierarchy, what would be the precedence rules ( or merging ) for service options that overlap ? ( Chaining -f does not look like real composition but not persisted inheritance )

The issue when chaining -f is that it is not persisted.

[ABitMoreDepth]

Whether or not docker supports extends in swarm mode is not even relevant to compose, is it?

If I have deploy statements in my configuration, compose warns me, then proceeds to ignore them when I'm using docker-compose up (which I do often when developing/testing).

Why should the above behaviour be any different if I'm using extends? Nothing would actually break from compose's perspective. Fair enough if docker-engine needs some extra logic to figure out whats going on but I don't really see why it should block composes' extends support, with the limited exception that it would be confusing to have a feature disparity.

Running development with three -f's would be a pain as well in my particular setup.

Finally, if compose is able to deal with extends support, it shouldn't be impossible to add support for that to docker-engine as well. If I had any time or experience with go I'd volunteer to have a go (EDIT: pun not intended...).

For the moment we've reverse engineered a python script to assemble a flattened compose file for use with docker-engine, and have left all our files in v2.1, however this makes it significantly more difficult to adjust deployment options as we go.

[willseward]

@ABitMoreDepth I 100% agree. Having multiple -f's is a pain. Supporting extends in Docker is the way to go.

[marascha]

+1 "Supporting extends in Docker is the way to go"
We make intensive use of extends in our stack definitions and were very disappointed to hear about it not being supported in docker 1.13.1 and docker-compose config (when executed on a v3 compose file).

[shin-]

Folks, as I wrote earlier, this is out of my hands at this point. Please comment on https://github.com/docker/docker/issues/31101 instead.

[shin-]

https://github.com/docker/cli/pull/569