Compose: When using multiple docker-compose.yml files from different directories, local paths are not followed correctly.

[brocoli]

Consider the following command, working from ~/myproj/:

docker-compose -f docker-compose.yml -f somesubmodule/docker-compose.yml up

Suppose somesubmodule/docker-compose.yml contains the following

services:
  myservice:
    build:
      context: ./myservice

docker-compose up fails because it will try to build myservice from ~/myproj/myservice/docker-compose.yml instead of ~/myproj/myservice/somesubmodule/docker-compose.yml.

The error message is something like: ERROR: build path ~/myproj/myservice/ either does not exist, is not accessible, or is not a valid URL.

This happened with docker-compose version 1.8.0-rc1, build 9bf6bc6, installed from Homebrew on my Mac OS X Yosemite version 10.10.5

Note that simply running docker-compose -f somesubmodule/docker-compose.yml up works fine, it's probably a glitch due to merging yml files and losing locality. Relative paths should be expanded before merging instead.

[lhm]

I'm having the same problem. It makes pulling together the docker-compose configs from multiple projects somewhat difficult. It appears as if only the first context is being used when combining multiple files:

$ pwd
/tmp/dc

$ tree
.
├── p1
│   ├── Dockerfile
│   └── docker-compose.yml
└── p2
    ├── Dockerfile
    └── docker-compose.yml

2 directories, 4 files

$ cat p1/docker-compose.yml 
version: '2'
services:
  p1:
    build:
      context: .
    command: echo 'hello from p1'

$ cat p2/docker-compose.yml 
version: '2'
services:
  p2:
    build:
      context: .
    command: echo 'hello from p2'

$ docker-compose -f p1/docker-compose.yml -f p2/docker-compose.yml config
networks: {}
services:
  p1:
    build:
      context: /private/tmp/dc/p1
    command: echo 'hello from p1'
  p2:
    build:
      context: /private/tmp/dc/p1
    command: echo 'hello from p2'
version: '2.0'
volumes: {}

$ docker-compose -v
docker-compose version 1.10.0, build 4bd6f1a

In the example above I'd expect that the build context of p2 be /private/tmp/dc/p2

I made a repo for it here: https://github.com/lhm/docker-compose-contexts-issue

[MrHash]

Is there any workaround for this problem?

[dpnova]

FWIW I have created an empty compose config at the root of my project:

version: "3"

Then make sure it's included all the time:

docker-compose -f ./docker-compose.yml -f ./somewhereelse/docker-compose.yml -f ./1/2/3/docker-compose.yml up

In doing this, you then use the same relative path for volumes and context in all the other compose configs through the project.

I'm going to create a command line wrapper for my team so they don't have to think about it too much.

[bgiorgini]

It didn't work for me adding the emtpy docker-compose.yml file in the root. Still the other 2 compose files get mixed up with the paths, taking the first relative path for all the other images to build.
Anyone else found another solution for this?

With the same sample files I got this:

➜  /tmp docker-compose -f docker-compose.yml -f ./p1/docker-compose.yml -f ./p2/docker-compose.yml config
services:
  p1:
    build:
      context: /private/tmp/p1
    command: echo 'hello from p1'
  p2:
    build:
      context: /private/tmp/p1
    command: echo 'hello from p2'
version: '2.0'

➜  /tmp docker-compose version
docker-compose version 1.14.0, build c7bdf9e
docker-py version: 2.3.0
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2j  26 Sep 2016
➜  /tmp docker version
Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:53:38 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:54:55 2017
 OS/Arch:      linux/amd64
 Experimental: true

[joeyhub]

I think there is a lot of confusion here with people wanting different things.

I want the behaviour described in the original ticket which should be possible to fix with a stub docker-compose file. some people don't want that behaviour. I have lots of compose files in subdirectory where I want the path to always be treated as from the root path for the project directory. There are limits on volume (not supporting relative) and I have some limits on symlink usage.

However with version 2 at least this doesn't appear to work (docker-compose.yml in base directory). It seems that the path is now enforced as being relative to the specific file loaded at least for build paths.

This is something where it would be great to give the user more control. It's currently inconsistent and hard to find appropriate documentation on.

For example with version 2, including a base configuration works for setting a new base path for extends.file but not for build.context. Except on investigation it turns out that it is working except that extends doesn't take the path from the first configuration file used.

I'm really pulling my hair out with this because it does all kinds of strange things automatically taking control over the user rather than giving the user control and visibility. It's not only this but things like not being able to extend VOLUME in the Dockerfiles and finding myself doing things like putting in dummy services just to build and tag an image. I find myself working around these tools more than working with them.

At least with extends gone (ironically a work around to have it work the other way around) I suppose that inconsistency is not an issue any more, however it would be nice to be able to specify the base directory rather via argument rather than having a dummy file for it.

[fabriziocucci]

I've found this link particularly clear when it comes to using multiple compose files.

I'm not sure whether the page has been published recently or whether the behavior described matches the current implementation but it _kind of_ makes sense.

Specifically:

Understanding multiple Compose files
...
When you use multiple configuration files, you must make sure all paths in the files are relative to the base Compose file (the first Compose file specified with -f). This is required because override files need not be valid Compose files. Override files can contain small fragments of configuration. Tracking which fragment of a service is relative to which path is difficult and confusing, so to keep paths easier to understand, all paths must be defined relative to the base file.

Although, I don't particularly agree that this should be _the only way_ of handling relative paths and I can try to elaborate a little bit more by providing a simple example.

Suppose you have the following:

|
|----> service-a (git submodule)
|          |
|          -----> docker-compose.yml
|
|----> service-b (git submodule)
|          |
|          -----> docker-compose.yml
|
 -----> docker-compose.yml

where:

1. service-a requires MongoDB to be stand up by itself, so its docker-compose.yaml should be a valid compose file, i.e. I should be able to cd service-a && docker-compose up, so paths should be relative to the service-a folder;
2. service-b requires Neo4j to be stand up by itself, so its docker-compose.yaml should be a valid compose file, i.e. I should be able to cd service-b && docker-compose up, so paths should be relative to the service-b folder;
3. the docker-compose.yml in my-project adds other services that should be used in combination with the docker compose files of service-a and service-b, i.e. I should be able to cd my-project && docker-compose -f docker-compose.yml -f service-a/docker-compose.yml -f service-b/docker-compose.yml up. In order for this to work, paths in the docker-compose.yml of service-a and service-b should be relative to my-project.

So it appears that you need to either settle for 1 and 2 or 3, but not all of them (unless you introduce some duplication). Maybe a simple solution would be to treat paths as relative to the docker-compose.yml file where they appear.

Is there a "nice" workaround to currently handle this scenario?

[dpnova]

You need to keep in mind that when you do a build compose sends stuff to the docker daemon, it's then outside of the context of compose. You should run compose with the verbose flag to see what the paths in your build section are being translated to when being sent to the docker daemon.

[gucki]

I just encountered the same problem. Why not introduce some prefixes with special meanings?

./bla -> uses ./bla relative to the current docker-compose.yml
!/bla -> uses ./bla relative to the first docker-compose.yml
~/bla -> uses ./bla relative to the project folder

[ZimbiX]

I ended up creating a docker-compose-wrapper that loads the docker-compose.yml files, using sed to make the context paths absolute.

This currently creates temporary files in dc/, but I'm wondering if I can use the -f - option to provide the files via STDIN using YAML's --- to separate the files.

#!/bin/bash

set -e

current_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd "$current_dir"

# docker-compose doesn't support preserving a compose file's relative context when you include it from another directory
# Workaround this by creating copies of the compose files with their contexts made absolute, and prepare the argument list of these files
docker_compose_files=(
  "$GOPATH/src/github.com/greensync/dex-services/docker-compose.yml"
  "$PROJECTS/dex-docs/docker-compose.yml"
  "$PROJECTS/dex-demo-ui/docker-compose.yml"
  "../docker-compose.yml"
)
docker_compose_command_args=(-p dex)
mkdir -p dc
rm -f dc/*
for i in "${!docker_compose_files[@]}"; do
  compose_file="${docker_compose_files[$i]}"
  compose_file_dir="$(dirname "$(readlink -f "$compose_file")")"
  sed "s|  context: |  context: $compose_file_dir/|g" "$compose_file" > "dc/${i}.yml"
  docker_compose_command_args+=(-f "dc/${i}.yml")
done

# Run the given docker-compose command, inserting the argument list of our modified compose files
set -x
docker-compose "${docker_compose_command_args[@]}" "$@"

You can see the resulting config using:

./docker-compose-wrapper config

Edit: Perhaps the best solution would be for docker-compose to keep the functionality of the -f <file> parameter for fragments, but add another parameter that allows you to include a complete docker-compose.yml file while preserving its relative context.

[joeyhub]

ZimbiX, docker compose I believe just wraps docker and some of those strings in the configuration will go all the way down through to the shell command.

That means that for volume you might be able to get away with something like:

volume: "${STANDARD_ENVVAR}/path:/container/mountpoint"

[ferrastas]

Having a project with multiple services

|
|----> service-a
|          |
|          -----> docker-compose.yml
|
|----> service-b
|          |
|          -----> docker-compose.yml

Modify the corresponding docker-compose.yml file replacing relative paths with an environmnent variable + default value something like ${SERVICE_A:-.} and ${SERVICE_B:-.}
version: '3' services: service-a: image: foo/bar volumes: - ${SERVICE_A:-.}:/code
Then in the root path an .env file with:
SERVICE_A=../service-a SERVICE_B=../service-b
And voilá, starting containers from root path will read the .env file and will replace it on-the-fly.
$ docker-compose -f service-a/docker-compose.yml -f service-b/docker-compose.yml config services: service-a: image: foo/bar volumes: - /Users/.../dev/sandbox/service-a:/code:rw service-b: image: foo/bar volumes: - /Users/.../dev/sandbox/service-b:/code:rw version: '3.0'

[GregYeo]

Temporary solution

You can use config command multiple times to avoid manipulating referencing docker-compose.yml file you already have.

Pros

• You do not need to change your current docker-compose.yml files

Cons

• It requires to make temporary files and clean them
• It requires to run docker-compose command multiple times

Example

docker-compose -f submodules/A_SUB_PROJECT/docker-compose.yml config > _tmp_.A.yml
docker-compose -f submodules/B_SUB_PROJECT/docker-compose.yml config > _tmp_.B.yml
docker-compose \
  -f _tmp_.A.yml \
  -f _tmp_.B.yml \
  config > docker-compose.yml
rm _tmp_.*.yml

[captain-kark]

@GregYeo I was able to get around the temporary files with this:

docker-compose -f submodules/A_SUB_PROJECT/docker-compose.yml -f submodules/B_SUB_PROJECT/docker-compose.yml config | docker-compose -f - up

You'll want to alias that command or put it make because you have to re-create that same docker-compose file and pass it in for commands like ps, down, or stop.

docker-compose -f submodules/A_SUB_PROJECT/docker-compose.yml -f submodules/B_SUB_PROJECT/docker-compose.yml config | docker-compose -f - ps

Edit: It's much better just to save the file and have a pre-run step where you build the docker compose file, and then use docker-compose commands as you would normally:

docker-compose --project-directory=. submodules/A_SUB_PROJECT/docker-compose.yml -f submodules/B_SUB_PROJECT/docker-compose.yml config > docker-compose.yml
docker-compose up -Vd
docker-compose ps

[dlgoodchild]

I've tried many of the solutions mentioned here, and I found that most did not work for me. The problem always seems to be that the path of the first docker-compose.yml file seems to be used in the other files (when using relative paths, i.e. beginning with ./), even though the path is clearly different.

I managed to get @ferrastas version working with a modification. I had to create a top level docker-compose.yml which contained only one line version: "3.7".
I then applied a top level .env as follows:

FRONTEND=./frontend/docker
BACKEND=./backend/docker
PROXY=./proxy/docker

In each of the directories listed in the .env file, I have a local docker-compose.yml which uses the variable syntax ${PROXY:-.} as part of a volume, a sample, ./proxy/docker/docker-compose.yml, being:

version: "3.7"

services:
  haproxy:
    image: haproxy:1.8.14-alpine
    volumes:
      - ${PROXY:-.}/haproxy/conf:/usr/local/etc/haproxy
      - ${PROXY:-.}/ssl:/usr/local/etc/ssl
    ...

I then execute this with:

docker-compose -f docker-compose.yml -f ./frontend/docker/docker-compose.yml -f ./proxy/docker/docker-compose.yml -p proj up

This answer is mostly a repeat of @ferrastas but with a minor tweak and with examples (thanks!).

[GanymedeIndustries]

I ended up using a bash script wrapper for building the config, but I do think that what @ferrastas and @dlgoodchild did with setting the environment is a more cleaner approach. My project was basically the same setup, where I have a frontend, a backend, a DB and a proxy wrapper for it all. To make my life easier, each "service" has their Dockerfile and docker-compose.yml files in their respected /docker/dev/ directories, and each relative path in docker-compose config is ../../ (context, volumes, etc). Here is an example of the backend config:

version: "3"
services:
  backend:
    image: backend:dev
    build:
      context: ../../
      dockerfile: docker/dev/Dockerfile
    ports:
      - "3000:3000"
    volumes:
      - ../../:/usr/src/app

Project setup is:

|
|--> /proxy
|---------> /dev/docker/Dockerfile and docker-compose.yml
|
|--> /backend
|---------> /dev/docker/Dockerfile and docker-compose.yml
|

You get the picture. Here is my script that lives in the root path:

#!/bin/bash

DOCKER_ROOT="${PWD}"
DOCKER_PATHS=(proxy frontend backend db)
DOCKER_CONFIG=""

for i in "${DOCKER_PATHS[@]}"; do
  CONFIG_ROOT="${DOCKER_ROOT}/${i}"

  CONFIG_TXT=$(cat "${i}/docker/dev/docker-compose.yml");
  CONFIG_TXT="${CONFIG_TXT//\.\.\/\.\.\//$CONFIG_ROOT}"

  # Need to remove the "version" and "services" lines on the non-first one
  if [ "$i" != "proxy" ]; then
    CONFIG_TXT="$(echo "$CONFIG_TXT" | sed /version\:/d | sed /services\:/d)"
  fi

  DOCKER_CONFIG=$(echo "${DOCKER_CONFIG}" && echo "${CONFIG_TXT}")
done

echo "${DOCKER_CONFIG}" | docker-compose -f - config

Notice that I remove the version and services lines from the config on the non-first config, this is because in the end I pipe it all the docker-compose config as one file instead of multiple -f files. As people have mentioned, the order of -f (in this case the array $DOCKER_PATHS) is very important, as the first config is your "base" and the rest are overrides. You could even do what @dpnova mentioned and just create an empty config as your base (or just hard code it here in the script) to make it more uniform.

[GanymedeIndustries]

For WSL users

If you are using WSL, then you need to specify all volumes as WSL paths, whereas all context paths should stay the same linux format. Below is a quick fix for the above script (plus a tweak using @dpnova advice):

#!/bin/bash

DOCKER_ROOT="${PWD}"
DOCKER_PATHS=(proxy frontend backend db)
DOCKER_CONFIG="version: \"3\"\nservices:"

for i in "${DOCKER_PATHS[@]}"; do
  CONFIG_ROOT="${DOCKER_ROOT}/${i}"
  WSL_CONFIG_ROOT=$(wslpath -au "${i}")

  CONFIG_TXT=$(cat "${i}/docker/dev/docker-compose.yml");
  CONFIG_TXT="${CONFIG_TXT/context\:\ \.\.\/\.\.\//context: $CONFIG_ROOT}"
  CONFIG_TXT="${CONFIG_TXT//\.\.\/\.\.\//$WSL_CONFIG_ROOT}"

  # Need to remove the "version" and "services" lines
  CONFIG_TXT="$(echo "$CONFIG_TXT" | sed /version\:/d | sed /services\:/d)"

  DOCKER_CONFIG="${DOCKER_CONFIG}\n${CONFIG_TXT}"
done

echo -e "${DOCKER_CONFIG}" | docker-compose -f - config

I'm only using context and volumes, so I can just replace context with one and the rest as the WSL path

[Gems]

I believe that docker-compose has a bug not respecting well the following part of the documentation:

CONTEXT - When the value supplied is a relative path, it is interpreted as _relative to the location of the Compose file_. This directory is also the build context that is sent to the Docker daemon.

A better bug description for reference: https://github.com/docker/compose/issues/3568

[ferrastas]

It's not a bug, it's clearly explained in https://docs.docker.com/compose/extends/

When you use multiple configuration files, you must make sure all paths in the files are relative to the base Compose file (the first Compose file specified with -f). This is required because override files need not be valid Compose files. Override files can contain small fragments of configuration. Tracking which fragment of a service is relative to which path is difficult and confusing, so to keep paths easier to understand, all paths must be defined relative to the base file.

[Gems]

Thanks a lot for pointing out to the corresponding documentation section.

[Gems]

I've created a docker-compose wrapper that changes the behavior in case of multiple files specified and makes each configuration file consider its own working directory.

[dlgoodchild]

For those interested, I've just tested @Gems docker-compose wrapper script (after changing the version from 3 to 3.7 within the script), and I have to say it has so far worked very well!

As a quick test, my command was:

$ chmod +x docker-compose.sh
$ ./docker-compose.sh \
    -f ./frontend/docker/docker-compose.yml \
    -f ./proxy/docker/docker-compose.yml \
    -p 016 up

I added in a project "name" (in this case a project number) as a test that it was recognised. My docker ps then looked like:

CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                                      NAMES
d657452408b7        haproxy:1.8.14-alpine   "/docker-entrypoint.…"   2 minutes ago       Up 2 minutes        0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   016_haproxy_1
cefc6fd95bf4        nginx:1.13.6-alpine     "/bin/sh -c 'envsubs…"   12 minutes ago      Up 2 minutes        0.0.0.0:8881->80/tcp                       016_nginx_1

Thought this might be useful for some folk who were maybe a bit unsure on how to use the wrapper.

A noteworthy mention is that if there is a .env file in the directory from where you run the docker-compose.sh file this will be picked up and used by docker-compose.

Thanks @Gems, it works a treat! I might move over to using that in favour of having to define and maintain env variables.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[leonklingele]

No

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.

[katsar0v]

Is there update on this matter? The documentation seems to be poor as well, nobody knows exactly how docker-compose behaves with multiple files

[katsar0v]

Well the solution says:

When you use multiple configuration files, you must make sure all paths in the files are relative to the base Compose file (the first Compose file specified with -f). This is required because override files need not be valid Compose files.

Why should all files be relative to the base file and not the corresponding docker-compose file? There are no variables supported, nor $(pwd) is possible. Hardcoding paths is not a good solution.

In my opinion paths should be relative to the path of the docker-compose file they are written in. Otherwise lots of confusion can happen.

Lets say I have a dir1 with docker-compose.yml and dir2 with docker-compose.extended.yml. If I run docker-compose -f docker-compose.yml -f ../dir2/docker-compose.extended.yml I need to make sure that the volume paths in docker-compose.extended.yml are like ../dir2/mounted-dir (for example) which is totally senseless. If I have ./mounted-dir:/mounted-dir as a volume in docker-compose.extended.yml it would search it in dir1 and not dir2. Unbelievably senseless..

Tracking which fragment of a service is relative to which path is difficult and confusing, so to keep paths easier to understand, all paths must be defined relative to the base file.

I don't agree there, a pretty simple logic could figured out like, if a relative path is defined in a file, it is relative to it, or it can search from base to all overwriting files/dirs...

Also a minor, the sentence:

This is required because override files need not be valid Compose files.

could be written better, clearer and maybe add some arguments/hints.

[hangyan]

@katsar0v Sorry , i thought this is a project of mine..

[katsar0v]

@hangyan It is currently working, so "technically" the issue might be closed. But IMO it could be done way better. I am not a docker developer, can just help with testing and documentation.

[ndeloof]

Support for multiple compose files was introduced to support overrides, which mean the 2nd and next files set by --file options _should_ be designed as complimentary configuration for the main compose file, not independent content with their own relative paths and resources.

Documentation need to clarify this behaviour indeed.

[myitcv]

Support for multiple compose files was introduced to support overrides, which mean the 2nd and next files set by --file options _should_ be designed as complimentary configuration for the main compose file, not independent content with their own relative paths and resources.

I think the point that many are raising here is that the semantics you describe are not (necessarily) exclusive with an approach that allows compose files from multiple projects to be composed.

For my own purposes, I've created a wrapper around docker-compose to address this issue and #7546.

[kmturley]

This an essential feature for a multiple-repo microservice architecture...

Using meta to pull in the code from multiple git repositories: https://github.com/mateodelnorte/meta
Then using docker-compose to run them all together.

Any idea on when we can get a fix for the paths issue? or official workaround?

EDIT: found a better way to do it, use a .env file containing:

COMPOSE_FILE_SEPARATOR=:
COMPOSE_FILE=./service-a/docker-compose.dev.yml:./service-b/docker-compose.dev.yml:./service-c/docker-compose.dev.yml
SERVICE_A=../service-a
SERVICE_B=../service-b
SERVICE_C=../service-c

Then you can use all the normal docker-compose commands without the additional -f arguments:

docker-compose config
docker-compose build
docker-compose up

I made a demo repo here:
https://github.com/kmturley/docker-compose-microservices

[bruno-oliveira]

Is there a solution for this?

A design document, or a better ticket describing exactly how or where to fix it?

I think this is really important.

In my case scenario, I have a monorepo with lets say 3 projects:

proj1, proj2, proj3

each of them with their own Dockerfile and docker-compose to run integration tests using testcontainers.

Now, inside a docker-compose for proj3, I want to reference the Dockerfile for proj1 to build a service.

What do I need to do such that, from a docker-compose in proj3, this works:

my-test-api-wtv:
       build: ../../../../whatever/folder/where/dockerfile/for/proj1/is

where inside this ecosystem can we see the exact place where this resolution happens?

With some pointers, I'd really be glad to help since this is partially blocking the IT setup we want to achieve at my $DAYJOB.