Compose: Usage of -f changes working directory for subsequent included files

[mikesir87]

Description of the issue

Usage of -f appears to change the working/base directory for subsequent files and causes those files to fail if using relative paths for build (or other) directives.

Context information (for bug reports)

Given two compose files at different locations...

/tmp/docker-compose.yml

version: "3.7"
services:
  db:
    image: mysql

[project-root]/docker-compose.yml

version: "3.7"
services:
  app:
    build: ./

and a Dockerfile at [project-root] (doesn't matter what's in it obviously), if, from the project-root directory, I run docker-compose -f /tmp/docker-compose.yml -f ./docker-compose.yml up, I get an error stating ERROR: Cannot locate specified Dockerfile: Dockerfile

But, if I swap the order of the files, the error doesn't occur and the compose stack starts. Unfortunately, this means that the file overriding isn't happening the way I'd like it to happen.

This is especially a problem when combined with the use case mentioned in https://github.com/docker/compose/issues/3440#issuecomment-421031181

Output of docker-compose version

docker-compose version 1.22.0, build f46880f
docker-py version: 3.4.1
CPython version: 3.6.4
OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018

Output of docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     true

Output of docker-compose config

When running with docker-compose -f /tmp/docker-compose -f ./docker-compose.yml, I get...

docker-compose -f /tmp/docker-compose.yml -f ./docker-compose.yml config
services:
  app:
    build:
      context: /tmp
  db:
    image: mysql
version: '3.7'

With the files reversed, I get...

services:
  app:
    build:
      context: /path/to/project/root
  db:
    image: mysql
version: '3.7'

Steps to reproduce the issue

(see above)

Observed result

The working directory has changed for other files, causing relative paths to fail.

Expected result

The working directory reverts back to where the command is being executed from for each file inclusion.

Cute animal for fun

[shin-]

Hi @mikesir87

Thanks for the report! However, this is the intended behavior and is not likely to change in the foreseeable future, as it would likely break existing applications that rely on it.

That said, you should be able to resolve the issue you describe by providing the --project-directory option to ensure relative paths are computed based on your [project-root]. Let me know if that helps!

[mikesir87]

That does help. I see now that the intended use of multiple compose files isn’t to deploy independent sets of services, but merging the output into a single “file” to be used for execution. Makes sense that that derived stack requires paths to be anchored at a single place, not potentially multiple roots (in the case both of the example files above had a relative path).

While I could easily accomplish my original goal with two separate compose commands (up, down, etc), it would be super convenient to have a single log stream. I guess in this case too, it doesn’t matter what order they are applied as they aren’t actually overriding each other, but simply adding non-conflicting services.

Thanks for the help! I will look through the docs to see if there might be a way to make this a little more obvious (may not be). If so, I’ll send a PR with an idea. 👍

[mikesir87]

Hmm... now that I've gotten on my computer again, I'm giving it a try using the --project-directory flag. But, it doesn't seem to be resolving the issue...

> docker-compose --project-directory $(pwd) -f /tmp/docker-compose.yml -f ./docker-compose.yml config
services:
  app:
    build:
      context: /tmp
  db:
    image: mysql
version: '3.7'

You'll see that services.app.context is /tmp, when I would have expected it to have been /path/to/project. In fact, the output doesn't change at all if I leave out --project-directory. Based on a few other experiments (swapping order, setting/not setting project-directory), it seems that the relative paths are always based on the location of the first compose file.

While I recognize the use case I mentioned in the original issue could still be done with two separate compose commands, here's another use case...

I'm using the output from docker-app render, but am wanting to override one (or more) of the services to use a build directive to produce a "dev" container. Due to #3440, I was rendering the output to /tmp, but then I run into the issue here. A workaround is to place the rendered output in the project root instead of /tmp. But, I don't want to confuse anyone on my team with the multiple compose files, so want to keep the docker-app rendered file "out of sight, out of mind." I hope that makes sense... (it's getting late now)

[shin-]

Thanks for following up! I'll look into it and reply more extensively once I know more, but this might very well be a bug.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.