Compose: It should allow extended services to define links

[docteurklein]

Hi!

I'm in a use-case where I have a basic docker-compose.yml file that defines a php service talking to a db service.

Now I added a second file named "dev.yml" where I extend all the services from docker-compose.yml and I add a volumes key to the php service.

This scenario does not work (services with 'links' cannot be extended) and I don't think the docs are quite clear on this behavor. (EDIT: it's clear but stay with me :))

Docs says that links won't be copied:

Compose copies configurations from the original service over to the local one, except for links and volumes_from. These exceptions exist to avoid implicit dependencies—you always define links and volumes_from locally.

What I'd like is to not being forced to create a prod.yml file, just to add a links section.
I'm perfectly OK with the idea of always defining links locally (and not merging the array like it's done with expose f.e).
But I think the base file can also be considered a perfectly valid leaf.

I'd like my docker-compose.yml file to be the standard, and dev.yml to add some volumes (and define links locally in it).

I hope I'm clear :)
I think there is no reasion why I can't define links in the base file. It would be great if I just had to redefine them if needed in any other file that extends it.

[BinaryMiscreant]

I run into this development scenario as well, wanting some dev.yml to just add a few things (volume, variables, etc. ) It would be nice to specify you want to inherit links in some way.

[docteurklein]

@BinaryMiscreant The idea is not to inherit the links, (you'll still have to redefine them at each extends).
The reason (and I think it's a valid one) is to be able to know the exact dependencies by reading only one file.

But my proposition is not about merging links with parent file, It's to let the parent file be able to define links on its own. And if a child overrides this parent, let it redefine all the links.

@aanand thoughts?

[dunk]

This seems overly restrictive. I have just come across this problem myself. On the one hand you have services all specified in one place, and on the other better code factorisation. I'd give people the choice, different circumstances will warrant different things.

[viranch]

+1 I agree. Extending links is seldom a destructive action anyway.

[mvallerie]

+1 for this.

[lucernae]

+1 for this.
From the documentation, it is clear that we have to define the links again locally. But compose doesn't accepts services with predefined links. I'm not against the idea of defining the links again. But, if I have to extend it from services which doesn't have links defined, then I end up having to create 3 yml configuration: common yml without links, production yml with links, and development yml with links. I'd rather not repeat myself in those configuration files.

A general use case why I need an extend features is because I want to expose a port of certain services without modifying its link in dev environment, or change some environment variables. Having to redefine the links actually feels complicating things.

[samuelololol]

+1 very agree!

As https://docs.docker.com/compose/extends/#reference says, links and volumes_from should be defined locally. As I know, the only way to extends those docker-compose.yml is to refactor them into common and link parts(maybe 2+ files) and use the common part to extend.

I am just wondering are there any shallow exdends to accomplish these things ?

[c0deaddict]

+1

[michaeljs1990]

kind of crazy that it doesn't just ignore links / volume_from and tell you that you must redefine it. This is a huge pain point and cases me to have duplicated docker-compose code all over the place that I must not manually watch and update.

Without supporting these features extend is almost completely worthless.

[aanand]

Thanks for your patience with this one, folks. Have a look at the new functionality added in #2051 - you can pass multiple files to Compose with -f, each acting as a file-level extends. Links etc. are allowed.

[michaeljs1990]

this is awesome @aanand. thanks for the info. hard to keep up with all the changes.

[samuelololol]

@aanand Is this override feature also applied in docker-compose.yml file for extends ? I mean will it settings override parents' service?

[aanand]

@samuelololol We need to nail down how extends and multiple -f args should interact. We have a few options:

1. Don't allow them to be used together. Nice and simple.
2. Fully resolve extends in every file as part of reading it in, _then_ go through the files in sequence.
3. Go through the files in sequence to produce a service->config mapping, allowing each file's extends keys to override the previous one's, _then_ resolve extends in each service that has it defined.

What's the intention of the following config?

docker-compose.yml

web:
  extends:
    file: common.yml
    service: foo

docker-compose.overrides.yml

web:
  extends:
    file: common.yml
    service: bar

To me, it looks like the user probably wants web to have (by default) all the config from bar, and _none_ of the config from foo. Which would be a case for option 3.

But I'm not sure anyone wants to do this in the first place. Which would be a case for option 1 - don't let people combine the features unless it turns out they want to, then figure out what they're trying to achieve, then decide if either of options 2 or 3 is therefore a good idea.

Would you want to combine them? How, and why?

[samuelololol]

Actually, what I want is extended configs in service->config mapping could be _replaced_ in extended docker-compose.yml. They can help me to reuse existing docker-compose.yml without spliting them into two separate parts, docker-compose-common.yml and docker-compose.yml. As an example described in this issue, links cannot be redefined in extended config.

2051 is a great feature to help modulize configs, but to me, it probably will not gracefully solve this case, I don't see helps to this by using -f argument to extends multiple docker-compsoe files. I would like to know the detail of extends in yaml and see if it's possible, but from your description it apparently not. :(

I still provide my thought to this. It is simple and easy as well. I would use extends in yaml as much as I could because the purpose of my using docker-compose is to reduce command line work of docker command, and I don't want it back again in docker-compose. And also, I don't see any possible yet to combine them together.

[dunk]

2051 is great, but I've been sharing the nagging suspicion that there's still something more to this. I think that what we're all describing, in a roundabout way, is template inheritance. In fact I almost started to write all my Dockerfiles in Jinja templates, until it dawned on me that there aren't enough hours in the day to shave _all_ the yaks. I imagine that docker-compose isn't the first, and won't be the last, system that requires this sort of pattern in its configuration files. Are there any good models out there for this? I lack the knowledge myself. But it feels like the sort of thing that has probably been solved before - and probably more than once. Template inheritance would be useful, but then if you follow that to its conclusion then eventually you're going to want some sort of analogue to multiple inheritance / mixins / whatever. And that could be painful. In my own system I have a cross-cutting concern - a syslog volume that is applied across many containers. It would be nice to express that in once place, but alas. Any thoughts?

[k0pernikus]

+1:

I have the use case of wanting to build our development stack without repeating the docker-compose.yml all over the place.

A project, ProjectA, provides a service that links multiple services.

This service is a dependency in ProjectB. So I thought I could just:

project-b:
    build: .
    links:
        - dependency

dependency:
    extends:
        file: ../project-a/docker-compose.yml
        service: project-a

Yet all I get is services with 'links' cannot be extended.

It seems I have to do a lot of copy pasting and manual labor in order to keep the docker-compose.yml-files in check.

[dnephin]

@k0pernikus I think #2051 might solve that problem for you. You can find the docs explaining its use here: https://github.com/docker/compose/blob/master/docs/reference/docker-compose.md (I'm working on more "user guide" style docs now).

[dunk]

TL;DR: I think that a mixin-type approach for docker-compose.yml services would solve a lot of problems.

There are often a number of different aspects that need to be factored separately in a system. Cross-cutting concerns (e.g. logging) are especially awkward, and lead to a lot of duplicated configuration code. As far as I can tell, neither using anchors and aliases in yaml files, nor the docker extends keyword will provide non-repetitive definitions of the kind I describe here. Therefore I think a deeper approach is needed.

Personally, I have resorted to generating my docker-compose.yml via Python (this code, included at the end, won't recurse up the class hierarchy properly atm, but it gives a concrete example of the features I'm talking about). I've included a subset of the containers in my system to illustrate. As you can see, I need all containers to have the SetTerm class applied, so that when connecting and interactively debugging I can have the TERM set to xterm-256color which a lot of docker images don't have by default, making anything interactive painful. Containers which subclass this may also define their own environment variables, and so these dictionaries must be merged. A large number of, but not all, containers need to be connected to the syslog volume. Containers which subclass this may also have their own volumes mounted, and so these lists need to be merged together. There are also some intermediate but not explicitly built classes like SampleTracking (no 'Container' suffix means it won't be rendered), which apply to a specific group of services.

Do others think that this would be a useful way to factor their services? And how might that best be achieved?

https://gist.github.com/dunk/8d258be750c7da048f52

[dnephin]

The new http://docs.docker.com/compose/extends/#multiple-compose-files feature allows extending a service with links and volumes_from

[michaeljs1990]

@dnephin Multiple compose files although helpful for some cases don't really solve the problem of composing micro services which seems to be a need from what I have seen in the issues here and on the #docker irc channel. Docker compose may not want to solve this issue but I have created a test implementation that lets you link docker-compose.yml files that have dependencies on each other. https://github.com/michaeljs1990/compress does docker-compose ever plan to support something like this?

[dnephin]

Have you seen #318 and https://github.com/dnephin/compose-addons ?

It's quite possible that it will be supported natively someday, but it adds a not of unrelated features, so it needs to be a feature that many people will use.

For now it can be implemented with the pre-processes and exports in https://github.com/dnephin/compose-addons

[michaeljs1990]

I have seen compose-addons but it seemed to not do quite what I wanted it to when merging files together. I had not seen issue #318 however.

[dnephin]

@michaeljs1990 cool, if you have suggestions for how it should work please do open an issue on the compose-addons repo

[scomma]

Hi, I think I've read every issue on this repo where a lot of people request that Compose doesn't refuse to extend containers which have the links directive. It can either extend the directive, or it could just complain and ignore the directive, but it should not flat out die.

I'm stuck on this issue as well, and I don't even have multiple docker-compose.yml files. I just have a bunch of Resque worker processes that work on different queues, and they all share the same basic configuration. So it makes no sense that you'd have to forego using extends just because they all link to a db container.

[kleptog]

I'm was disappointed when I ran into this issue today. I wanted to create a test setup with Selenium by taking a few of the containers from the production compose file and adding a few for Selenium. Nope, it breaks on the links in the production file.

It would be more friendly if the error message said something like:

ERROR: Cannot extend service 'service' in docker-compose.yml: services with 'links' cannot be extended, please specify 'links' again

And that I could then specify the needed links again in the new file. I've ended up copying the definitions, but I know already they're going to get out of sync with the production file :(

[michaeljs1990]

@kleptog any reason using external_links does not work for you?

[kleptog]

External_links is for linking to containers already running elsehwere. That solves half the problem, I could run two different compose files. That doesn't change the fact I want to use a subset of the production file with slightly tweaked parameters (mostly the same environment variables, different volumes).

[dnephin]

I suggest using multiple compose files, it is designed to solve this problem.

[kleptog]

You are suggesting that I take my single production compose file and split it into several pieces so different places can use different parts and none can be used alone because they need to link to each other? And then create scripts to wrap docker-compose for all the different combinations, because otherwise no-one can figure out how to start the complete application? I thought the goal was to make things clearer...

I think I'd prefer generating the compose files using a templating language to that.

[jtmarmon]

+1 i have the exact same use case as @scomma. We also have workers for message queues. we have like 5 workers and I have to repeat the same configuration for all of them. it looks horrific, especially at 16 containers.

I think extending prod to add some dev configuration is solved decently by using separate files, but I just have a single dev file (we're not using compose in production) and want to be able to run the same codebase with multiple commands (rake resque:scheduler, rake resque:work, shoryuken, rails s, etc). Having to redefine the entire thing just because they all use links (for which inheritance would work fine) doesn't make sense to me.

If either ignoring or permitting inheritance isn't preferrable, why not add a force_inherit_links or something as happy medium

[willejs]

+1 on this

[atrauzzi]

+1

[sashgorokhov]

I really want this feature too! +1

[LouisKottmann]

The config splitting is definitely not satisfying, I should be able to tell that I know what I'm doing and I want to specify new links.

I have several apps that all have a docker-compose config (itself extending bases), and then a master docker-compose config that is meant to run them all together, but linked to a common database.

Imagine the pain it is for me to maintain dozens of docker-compose files that are split...

[dunk]

@LouisKottmann I have been puzzling over this (and, in fact, the larger case) for a long time myself. There a some discussion in #3167 which is relevant.

[mjsisley]

+1

[stefanfisk]

I find that using multiple files with -f works OK, but would it be possible to add a root level extends: to the composer files such that the dependencies can be specified within the file instead of via the command line?

[dijkstraj]

+1

[neronmoon]

👍

[abrahamrkj]

+1

[ppowalowski]

+1

[lpastusz]

+1

[christhomas]

Hi,

I have a pretty solid use case for this, I have the following configuration, look at the almost complete duplication here, I have 15 nginx containers in total, one for each country code, I am only showing a couple here as an example:

  mysql:
    build: docker/mysql
    ports:
      - "10002:3306"

  phpfpm:
    build: docker/phpfpm
    volumes:
      - $PWD/projects/src/:/www/

  nginx_ae:
    build: docker/nginx
    links:
      - nginx-proxy
      - mysql:database
      - phpfpm:php
    volumes:
      - $PWD/docker/nginx/symfony.conf:/etc/nginx/conf.d/default.conf
      - $PWD/projects/src/:/www/
    environment:
      - VIRTUAL_HOST=ae-dev.company.local

  nginx_bd:
    build: docker/nginx
    links:
      - nginx-proxy
      - mysql:database
      - phpfpm:php
    volumes:
      - $PWD/docker/nginx/symfony.conf:/etc/nginx/conf.d/default.conf
      - $PWD/projects/src/:/www/
    environment:
      - VIRTUAL_HOST=bd-dev.company.local

  nginx_co:
    build: docker/nginx
    links:
      - nginx-proxy
      - mysql:database
      - phpfpm:php
    volumes:
      - $PWD/docker/nginx/symfony.conf:/etc/nginx/conf.d/default.conf
      - $PWD/projects/src/:/www/
    environment:
      - VIRTUAL_HOST=co-dev.company.local

  nginx_id:
    build: docker/nginx
    links:
      - nginx-proxy
      - mysql:database
      - phpfpm:php
    volumes:
      - $PWD/docker/nginx/symfony.conf:/etc/nginx/conf.d/default.conf
      - $PWD/projects/src/:/www/
    environment:
      - VIRTUAL_HOST=id-dev.company.local

..... + 11 extra practically configurations

All of those nginx containers are duplicates, apart from the VIRTUAL_HOST which allows the nginx-proxy to reverse proxy port 80 to the correct container based on the VIRTUAL_HOST parameter, all of this duplication happens because I cannot inherit links or volumes

This is all in the same file, not multiple files, so the logic about forcing people to make visible things from other files, doesn't hold here, with inheritance, I could do something like this:

  mysql:
    build: docker/mysql
    ports:
      - "10002:3306"

  phpfpm:
    build: docker/phpfpm
    volumes:
      - $PWD/projects/src/:/www/

  nginx:
    build: docker/nginx
    links:
      - nginx-proxy
      - mysql:database
      - phpfpm:php
    volumes:
      - $PWD/docker/nginx/symfony.conf:/etc/nginx/conf.d/default.conf
      - $PWD/projects/src/:/www/

  nginx_ae:
    extends: nginx
    environment:
      - VIRTUAL_HOST=ae-dev.company.local

  nginx_bd:
    extends: nginx
    environment:
      - VIRTUAL_HOST=bd-dev.company.local

  nginx_co:
    extends: nginx
    environment:
      - VIRTUAL_HOST=co-dev.company.local

  nginx_id:
    extends: nginx
    environment:
      - VIRTUAL_HOST=id-dev.company.local

Much nicer and totally makes sense.

[mtomas]

+1

[kala888]

+1

[mo-mughrabi]

+1

[herriojr]

+1

[inwaar]

+1

[weeger]

+1

[eduardohenriquearnold]

+1

[shin-]

Extension fields should address the remaining pain points on this issue. Please let me know if there's something I missed!

[axsuul]

@shin- Great! But I believe you meant this link instead? https://docs.docker.com/compose/compose-file/#extension-fields

[shin-]

I usually link to the v2 reference because it's better suited to work with docker-compose (whereas v3 is better suited for docker stack). In this instance, the content is the same, so it doesn't matter :)

[scmorse]

How do extension fields let you get around not being able to extend a service with links?