Compose: Overriding the set of values when extending the "ports" option.

[ghost]

The default behavior when extending a service or overriding a whole docker-compose.yml file is to concatenate the sets of values on the multi-value options: ports, expose, external_links, dns and dns_search.

The problem appears when for example you have two production hosts both serving on the port 80 and a single machine for development, there is no way of extending or overriding the services for a development environment that avoids the port collision on the host.

An example: lets assume there are two servers one is a REST API and the other is a webpage that uses the API, both listen on port 80 on the production host

• compose.yml on host A:

restserver:
  image: node
  ports:
    - "80:80"

• compose.yml on host B:

webserver:
  image: django
  ports:
    - "80:80"

On a development environment where you want to test this setup you would define:

• compose.override.yml on dev for A:

restserver:
  ports:
    - "127.0.0.1:3030:80"

• compose.override.yml on dev for B:

webserver:
  external_links:
    - "service_restserver_1:api.restserver.domain.com"
  ports:
    - "127.0.0.1:8080:80"

Of course the "ports" option is concatenated with the production and there is no way of running both containers on the same host because both try to bind to port 80.

Is there any workaround or yml specific syntax for this use case?

[dnephin]

I would do something like this:

docker-compose.yml (no ports)

restserver:
  image: node

webserver:
  image: django

For dev you can setup some host ports

docker-compose.override.yml

restserver:
  ports: ['3030:80']

webserver:
  ports: ['8080:80']

and for prod you can set them to a different port:

docker-compose.prod.yml

restserver:
  ports: ['80:80']

webserver:
  ports: ['80:80']

In prod you would run docker-compose -f docker-compose.yml -f docker-compose.prod.yml up

[ghost]

It seems like a good, clean approach. Only cons would be the verbose command on production but that one is automatic.

Thanks @dnephin

[gregmartyn]

Another downside is that it no longer "just works". The user has to either create an override file or specify ports manually.

It would be nice if the override file could actually override array values like it does for scalar values. Instead, the override file appends array values and only actually overrides scalar values.

[mkurzeja]

I agree with @GregMartyn . A feature that would allow to override ports array instead of merging them would be awesome.

In my case we use it to override a port when one of the developers is not able to work on the default one.

Maybe docker-compose could just override the ports that are already forwarded: If port 80 is specified in both files, it uses just the one from docker-compose.override.yml?

[udhayakumaran]

@GregMartyn @mkurzeja Docker is already having that feature. I tried that to override with simple docker-compose, it is working.

For example,

docker-compose.yml with content,

my-httpd:
  image: httpd:latest
  ports:
    - "1110:80"  

And docker-compose.override.yml with content,

my-httpd:
  image: httpd:2.4

This is the docker ps info,

12286c55b76b httpd:2.4 "httpd-foreground" 5 seconds ago Up 4 seconds 0.0.0.0:1110->80/tcp dockercomposetest_my-httpd_1

It is taking override's image and port from the compose file.

[gregmartyn]

@udhayakumaran see above. "the override file appends array values and only actually overrides scalar values."

The image in your example is a scalar value and did get overridden. The port you specified is part of an array and was not overridden. You didn't specify any ports in your override file, so they were not overridden. Had you specified ports in the override file, they would have been combined with the port specified in the main config file.

[udhayakumaran]

@GregMartyn Okay, I understand now. Thanks for correcting me..

[datacarl]

@GregMartyn If you are using the .env file introduced in 1.7 you can define the compose files there. You'll have to ensure you have the correct .env file (a prod version for prod, dev version for dev) ofc but that's the only manual change required.

For dev environments you can set
COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml
and for production
COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml

Together with what @dnephin suggested it works fine.

[kevin-cantwell]

@datacarl Do you have a link to documentation on the DOCKER_FILE env var? It's not clear to me how it's used by Docker Compose.

[datacarl]

@kevin-cantwell https://docs.docker.com/compose/env-file/

(should be COMPOSE_FILE, my bad. Edited my reply above)

[kevin-cantwell]

@datacarl Ah, COMPOSE_FILE. Found it now, thanks: https://docs.docker.com/compose/reference/envvars/#compose-file

[CarstenHoyer]

I have added a preliminary pull request, that introduces a way to add an overwrite strategy other than the default append/union. The overwrite config currently only supports "ports" but it would be a smallish thing to support the other array options too.

Your compose file would then look like either:

version: "2"
services:
  web:
    override_strategy: overwrite

or

version: "2"
services:
  web:
    overwrite:
      - ports

And the result could be that if docker-compose.yml has ports: 80, 443, and your override file has 8080 and 8443 fx, then only the 8080 and 8443 ports would be in the final config.

[dimaqq]

@CarstenHoyer did your overwrite: change make it to any release?

[CarstenHoyer]

@dimaqq - I am afraid not. I have not kept the PR up to date with the upstream, and it also doesnt contain any automated tests.

Since I never got any official feedback, I figured it was not in scope.

[im-n1]

Just ran into thisand it's annoying. Compose file should act consistently so if we can override scalar values then we should be able to override arrays too.

Maybe, just maybe, there could be nifty simple trick from template inheritance all across most of web frameworks...

When you want to extend not override you can include parent value into child. Watch this.

Parent:

    ports:
        - 9000:80

Child:

    ports:
        - ROOT
        - 9001:81

Now children has both ports mapped because we included ROOT's value to the overriding value.

[TheTedAdams]

Adding another voice that this behavior is unexpected and unwelcome. Everything else in the override file... overrides. Then I tried to override the port and it just mapped both of them. Everyone on my team was surprised, as this is not expected behavior from the override file...

[shcallaway]

Two years later... Is this feature even up for consideration?

[achton]

This PR was closed way back in 2015 by @mnowster (who it appears is no longer contributing to this project) because the issue originally appeared to be solvable via configuration - and has been closed since.

However, based on recent comments from people who share this problem, I think it could do with a renewed vetting by a team member like @albers or @shin- who could weigh in on whether this is an unwanted fix, or if the issue may have some merit to it (in which case someone can work on completing the fix).

In the latter case, I recommend reopening this issue to get more eyes on the proposed change by @CarstenHoyer in Sep 2016 - there is no longer a accompanying PR, but a diff is available here. There is a PR here: #3939

[shcallaway]

@achton I opened a new PR with some of the requested functionality. #5354

[mallchin]

Run into this issue and am surprised it works this way -- it would be more logical to have child options squash parent ones.

[gtirloni]

Just spent some time trying to understand why it was not overriding the list of ports and found this issue.

[garyo]

I also would like "docker compose up" to just work while setting different ports for dev (override). The workarounds in this issue work, but will confuse others if I use them because the standard way to do things no longer works.

[nyetwurk]

There was no good reason to close this.

[mallchin]

Has this been resolved or should we open a new issue?

[nyetwurk]

It has NOT been resolved. The workaround cited above involves a huge mess of override files, most of which have duplicated entries if you have a docker-compose.yml file with many port directives that are override-able. The ONLY place you can make a customizable (and optional, such that it can be omitted from source control and not .gitignored) file is .env, which can only set COMPOSE_FILE (whose default is docker-compose.override.yml

This then ends up being the ONLY file you can put sane defaults in, which prevents the end user from using docker-compose.override.yml for other things.

Please reconsider PR #5354 or PR #3939

[at0ms]

+1

[im-n1]

aaaaand not a single fcuk was given that...um 3 years?

[tommyalatalo]

I also concur that the override behavior should be revised, arrays should be overwritten, not appended. Having to split fully working docker-compose.yml files and update submodules for port overrides etc to work is not reasonable.

[nyetwurk]

I actually disagree, since most of the time overrides are used to add to arrays (e.g. lists of things to do). When you do need overrides to overwrite, the target is usually dictionary k/v pairs. "ports" is just odd; it should have been a dictionary to start with.

[keithy]

The file is called composer-override.yml not composer-merge.yml. the only thing you can't do is override!
The net result is that there is actually no mechanism for overrides!

If environment: is defined, adding an env_file doesn't override anything either.

I don't actually get it. If you want a base that works out of the box, and then an additional config for different scenarios, that simplest of use case isn't supported. Yet the whole design of docker files is based on the concept of down load and derive something new without touching the layers above.

One solution would be, named ports.

[nyetwurk]

@keithy agreed completely about the simplest use case: establish upstream defaults, allow a minimal way to itemize only the relevant changes in a local override, on downwards for all derived layers.

The way "ports:" is designed (as an array) makes this impossible. If it was a dictionary, that hierarchy of overrides could be done easily (as long as there isn't the oddball case of wanting to map a single internal port to multiple external ports)

[keithy]

Kludge solution...

Wrapper called - docker-composer
Mark the ports to be overridden #overridable!

!/bin/sh

grep -v '#overridable!' docker-compose.yml > .tmp-dc.yml

docker-compose -f .tmp-dc.yml -f docker-compose.override.yml $@

rm .tmp-dc*

[ggirodda]

One solution to this problem is to set the ports only in override file. You can have a docker-compose.override.yml.exemple in your repository which will be renamed in docker-compose.override.yml the first time, so you can edit ports (or not) without touching the docker-compose.yml file

[c9s]

god... in order not to override the host port, I have to create 2 more config files ?

[nyetwurk]

Yes, the workarounds make no sense

[vectorjohn]

Hey, YAML itself already solved this problem with anchors and references. It allows you to extend or override as part of the YAML format itself. Maybe instead of a home rolled system, docker-compose could use that. I know, I should submit a PR and do it. But just throwing this out there in the mean time.

[nyetwurk]

Again, a simple dictionary (key/value) would fix this trivially. The order is just wrong. Even more obvious if you need to bind to an IP address.

I have no idea why this is closed

[sgohl]

This problem is even bigger, when you need different volume definitions and have to use multiple files. YAML merging/anchors are not sufficient. I need services for developers, and the same service with different definitions for developers of other departments, quality assurance, testing ... I need to override volumes. (Because: A developer of serviceA needs the repository mounted into the container. But QA just needs the ci-built-image containing the published code, which must not have a volume mount)

There should be a possibility to merge definitions. Or can you put an example how to use an anchor of another docker-compose.yml file? Even if that would be possible, it would make the additional file senseless and specific persons would have to modify the docker-compose.yml or an .override. file themselves, which has to be avoided. I want to serve a main yml and different override yamls for different persons

[keithy]

Solution available see http:gitHub.com/keithy/sprankle-podOn 3 May 2019 11:27 am, port22 [e-mail] wrote:This problem is even bigger, when you have to use multiple files. YAML merging/anchors are not sufficient. I need services for developers, and the same service with different definitions for developers of other departments, quality assurance, testing ... I need to override volumes.
There should be a possibility to merge definitions. Or can you put an example how to use an anchor of another docker-compose.yml file? Even if that would be possible, it would make the additional file senseless and specific persons would have to modify the docker-compose.yml or an .override. file themselves, which has to be avoided. I want to serve a main yml and different override yamls for different persons

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[nyetwurk]

Does anybody else see why I am suggesting a dictionary rather than an array?

[mike-code]

I do but the compose project looks semi abandoned. I wouldn't recommend it using in production.

[im-n1]

@mike-code thats nonsense this project is quite active.

[vectorjohn]

I don't think the dictionary suggestion is a good one, docker-compose shouldn't have some new half baked scheme for merging files when YAML already has a standard that does all that is needed (other than @keithy 's use case, which sounds so complex it's out of scope).

[nyetwurk]

Portmapping is fundamentally a dictionary, not an array, and merging comes for free.

[im-n1]

@vectorjohn is there any official yaml lib that can do the merging?

[keithy]

My use case and solution is very simple actually:

docker-compose allows you to supply .env as an environment file. The environment variable COMPOSE_FILE provides a list of compose_files to be assembled.

In those individual files I place environment variables where there are ports mentioned, so that the .env file can supply those values also.

The missing piece of the puzzle, as with so much of the docker stuff, is to make the .env file readable. I do this by creating the .env file from a readable version using an all-to-simple pre-processor docker-compose-use

[vectorjohn]

@vectorjohn is there any official yaml lib that can do the merging?

Oddly enough, docker-compose already supports this it looks like:
https://medium.com/@kinghuang/docker-compose-anchors-aliases-extensions-a1e4105d70bd

[typoworx-de]

How does the override feature work for version: '3.6' in yaml head? Trying to override some settings in DDEV-Project (https://github.com/drud/ddev) also using docker-compose (version 1.24.1 which seems to be latest release already).

Adding override_strategy: overwrite won't work and breaks docker-compose:

Unsupported config option for services.web: 'override_strategy'

[mcabrams]

@typoworx-de That was a proposed solution with a corresponding pull request that did not get merged. It is not available in version 3.6, or any other version for that matter.

[noahtallen]

One reason this is very important (didn't see it mentioned yet) is that we don't always have access to change the base docker-compose.yml file. In fact, I'm using a dockerfile provided by a different service that I cannot control. I probably ought not to update that file since it is in the source control of that other service.

If I was able to create env files and remove ports from the base config, then sure, I'd accept that there's a workaround. But you aren't always in control of the base config file, and modifying it directly generally isn't the best idea.

[nyetwurk]

I'm also unclear how you could use yaml array merge to remove an entry from the list, whereas this is trivial if it is a dict

[ebuildy]

What about adding a excludes option in extends clause?

service_A:
   extends:
     file: docker-compose.yaml
     service: service_A
     excludes: ["image", "ports"]
   build: .

[kkom]

@ebuildy - looks quite promising to me, but note that extends is not available in Compose file versions 2.1 and up: https://github.com/moby/moby/issues/31101

[lewcianci]

This should be reopened as the existing functionality is inconsistent with scalar values being overwritten by default. Or people should be able to make a choice - it makes no sense for them to be appended by default.