Compose: Multiple Compose files with common anchors do not work

[mitsos1os]

Using the anchor utility of YAML, we cannot use some common defined anchors (such as extension fields, x-*), to declare common configuration in the base compose file and then use them in the extra compose overrides. Docker compose throws error found undefined alias

To reproduce create fragment in docker-compose.yml:

x-function: &json-logging
  driver: "json-file"
  options:
    max-size: "100m"

And use this anchor in a point in a docker-compose.prod.yml:
logging: *json-logging

Running docker-compose -f docker-compose.yml -f docker-compose.prod.yml config will get the error mentioned. Only solution is to duplicate information across all compose files

[shin-]

Yes, anchors are consumed when the JSON file is parsed - this is working as intended.

[alexandernst]

So, anchors are basically not suitable to be used in multiple files? @shin- ?

[shin-]

Yes, they can't be used across different files.

[marioprudhomme]

Is this feature considered in future releases?

[kitzilla]

I was asked to split down my docker-compose.yml to multiples and this lead me to totally abondone extension fields. It would have been nice if the yaml inherit/override anchors from yamls in the left hand side of -f options.

[rulatir]

This is not "working as intended", this is a severe deficiency.

[gautaz]

@rulatir
Anchors are part of the YAML specification which states:

A YAML character stream may contain several documents. Each document is completely independent from the rest.

So this issue (if it is even one) has nothing to do with Docker but with YAML.

[mcabrams]

Are there any recommended ways of achieving the original goal here? I.e. reusing/extending configuration in another docker-compose yaml file? I'm aware that services can be added, but curious about doing things like extending an existing service.

[gautaz]

@mcabrams You might want to call docker-compose with multiple YAML files like this:

docker-compose -f a.yml -f b.yml

b.yml will extend/modify what was declared in b.yml.

This is documented here:
https://docs.docker.com/compose/reference/overview/#specifying-multiple-compose-files

[mcabrams]

@gautaz Thanks -- I'm familiar with and currently use extension by specifying multiple compose files. My question is whether anyone has any strategies for achieving extension via multiple compose files and reusing common configuration of services.

[rbdiang]

@mcabrams Have you found an alternative strategy? I had a similar plan as you, storing all of my common volumes, secrets, and configs as yaml snippets using anchors in a compose.base.yaml file with aliases in my compose-stack.yaml as in:

 docker-compose -f compose.base.yaml -f compose-app.yaml config > app-stack.yaml

I hear what @gautaz is saying, but IMO without cross compose file yaml anchor support the idea of building up a compose file from multiple yaml files isn't very useful 😞.

[Aradiv]

I had the same issue
my solution is instead of using
docker-compose -f docker-compose.yml -f ci/docker-compose.prod.yml build
I'm using
cat docker-compose.yml ci/docker-compose.prod.yml | docker-compose -f - build

This resolves the the aliases inside docker-compose.prod.yml correctly.

There is only 1 small tweek needed

docker-compose.yml

services: &default-services

ci/docker-compose.prod.yml

services:
  << : *default-services

[AdrienPoupa]

This is a limitation of PyYAML, not the YAML format in general.

I think this would work for PyYAML: https://github.com/ChloeTigre/pyyaml-keep-anchors

There also exists an extension for Ruamel.YAML that enables this feature:

https://stackoverflow.com/a/55975390/11115846
https://github.com/Tristan-Sweeney-CambridgeConsultants/ccorp_yaml_include

[newsages]

Normal, PyYAML,
Py-People are very garbage cans