Compose: Relative paths for -f are broken, and --project-directory appears to be stillborn

[Robin479]

Our code-project is structure such that all the docker related stuff resides in a sub-folder ./docker including ./docker/docker-compose.yml and various files and folders (e.g. ./docker/nginx/nginx.conf) to be volume-mounted by docker-compose using relative paths.
Our build job used to invoke docker-compose from the project root (e.g. /workspace/project) with the -f ./docker/docker-compose.yml option, which started to fail since the update to 1.12.0, complaining that it couldn't find /workspace/project/nginx/ for mounting, which clearly should have been /workspace/project/docker/nginx/. I can change directory to /workspace/project/docker to avoid using the -f … option, to work around the problem. However, this clearly is a bug introduced in recent docker-compose, since nothing has changed in our project except the update to 1.12.0 (from 1.11.0, I guess).

ERROR: build path /workspace/project/nginx either does not exist, is not accessible, or is not a valid URL.

After some research I found, that 1.12.0 introduced the --project-directory option, so I tried docker-compose --project-directory ./docker from the project-root /workspace/project as before. It doesn't appear to work either, complaining that it couldn't find a suitable docker-compose.yml in _this directory_. Presumably, _this directory_ mentioned is falsely assumed to be /workspace/project (=PWD), and docker-compose fails to properly check /workspace/project/docker as instructed (or maybe, I misunderstood the intended function of --project-directory …, in which case the -f … bug still applies).

ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?

    Supported filenames: docker-compose.yml, docker-compose.yaml

Using absolute paths as in docker-compose -f /workspace/project/docker/docker-compose.yml or docker-compose --project-directory /workspace/project/docker won't work either, except iff PWD=/workspace/project/docker, which renders -f and --project-directory completely useless.

I shall add that all mentioned paths are on the same file-system, and there are no soft-links involved.

For now, the only way to work around these problems is to cd /workspace/project/docker before every (parameterless) docker-compose invocation.

[dazhao-msft]

We have got users reporting the same issue: https://developercommunity.visualstudio.com/content/problem/40763/docker-compose-using-incorrect-path.html

[shin-]

Thank you for the report, we'll take a look ASAP.

[shin-]

I tried a simple reproduction of this based on your report but I can't seem to be able to reproduce. Here's what I'm doing:

$ tree
.
└── docker
    ├── docker-compose.yml
    └── nginx
        └── nginx.conf

2 directories, 2 files

$ cat docker/docker-compose.yml 
version: '2.1'
services:
  foo:
    image: nginx
    volumes:
      - ./nginx/:/conf
    command: cat /conf/nginx.conf

$ docker-compose -f docker/docker-compose.yml up
Starting docker_foo_1
Attaching to docker_foo_1
foo_1  | foo
foo_1  | bar
docker_foo_1 exited with code 0
$ docker-compose version
docker-compose version 1.12.0, build b31ff33
docker-py version: 2.2.1
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016

Did I miss something? Are you able to share your docker-compose.yml so I can gather more info?

Thanks.

[Robin479]

It appears the error message is related to the build: ./nginx setting:

$ pwd
/home/kk/workspace/docker-compose-fail

$ tree
.
└── docker
    ├── docker-compose.yml
    └── nginx
        └── Dockerfile

2 directories, 2 files



md5-77c121f874126fb5788e97900fcf2eab



$ cat docker/docker-compose.yml 
version: "2.1"

services:
  nginx:
    image: shd/nginx
    build: ./nginx



md5-77c121f874126fb5788e97900fcf2eab



$ cat docker/nginx/Dockerfile 
FROM nginx:alpine

MAINTAINER SHD System-Haus-Dresden GmbH



md5-77c121f874126fb5788e97900fcf2eab



$ docker-compose -f docker/docker-compose.yml config
ERROR: build path /home/kk/workspace/docker-compose-fail/nginx either does not exist, is not accessible, or is not a valid URL.



md5-77c121f874126fb5788e97900fcf2eab



$ docker-compose version
docker-compose version 1.12.0, build b31ff33
docker-py version: 2.2.1
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016

[Robin479]

After removing the build: ./nginx setting and adding a volume mount, I have this:
```$ cat docker/docker-compose.yml
version: "2.1"

services:
nginx:
image: nginx:alpine
volumes:
- ./nginx/:/conf

And the error message is gone (the error is not, though):

$ docker-compose -f docker/docker-compose.yml config
networks: {}
services:
nginx:
image: nginx:alpine
volumes:
- /home/kk/workspace/docker-compose-fail/nginx:/conf:rw
version: '2.1'
volumes: {}
```

Note the erroneous path translation on the volume mount, which should have been /home/kk/workspace/docker-compose-fail/docker/nginx:/conf:rw, but it is missing the docker/ sub-path.

[shin-]

Thank you for taking a closer look. I have a patch for this in #4713 if you're able to check it out and verify it works for you as well.

[dazhao-msft]

@shin- Do you have private bits with the fix somewhere? I'd like to try it out as well.

[shin-]

@dazhao-msft For Windows, you can download the Appveyor artifact here: https://ci.appveyor.com/project/docker/compose/build/master-3146/artifacts

[dazhao-msft]

I've verified that the issue reported at https://developercommunity.visualstudio.com/content/problem/40763/docker-compose-using-incorrect-path.html also gets fixed by the change. Thanks for the quick turnaround!

[dazhao-msft]

@shin-
Joffrey, do you know when the fix will be integrated in Docker CE for Windows? I checked the latest version 17.05.0-ce-rc1-win8 (11189), and the fix is not there yet.

This issue completely blocks the users from using Docker CE for Windows with Visual Studio. We hope to get this fix in ASAP.

Thanks!
-David

[STDGA]

Hi @shin,
Thanks for the fix, after i have downloaded this binary image: https://ci.appveyor.com/project/docker/compose/build/master-3146/artifacts
and copied it to "C:Program FilesDockerDockerresourcesbin" , it works again for me. I'm using Visual Studio 2017 with docker support and i'm able to debug it again.

Thanks,
Ghodrat

[friism]

@dazhao-msft this is targeted for 17.05!

[paulkling]

The fix from above fixed my problem as well.

[dlgoodchild]

I'm using Docker Toolkit for macOS, and I'm seeing this exact problem... but I'm on 17.06? and I'm up to date according to:

Docker 17.06.2-ce-mac27 is currently the newest version available.

$ docker-compose -v
docker-compose version 1.14.0, build c7bdf9e
$ docker -v
Docker version 17.06.2-ce, build cec0b72

What's the status of this issue with macOS?

Cheers

[shin-]

@dlgoodchild The stable branch of Docker 4 Mac doesn't always include the latest version of Compose. While the issue reported here was fixed in 1.13, you might be running into another bug with similar symptoms. Please try with Compose 1.16.1 and open a new issue if you see the problem persist.

[nik131186]

Not able to fix the below error. Any help in this matter is really appreciated.

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c curl http://keplerproject.github.io/luarocks/releases/luarocks-2.2.2.tar.gz -O && tar -xzvf luarocks-2.2.2.tar.gz && cd luarocks-2.2.2 && ./configure && make build && make install' returned a non-zero code: 2

I tried to install unp on Mac so that it can support (tat.gz) file. But no luck so far

[Robin479]

@nik131186

1. Your question is totally unrelated to this issue, i.e. off-topic!
2. Your command should probably be quoted properly like this (please note where I put the single and double quotes): /bin/sh -c 'curl "http://keplerproject.github.io/luarocks/releases/luarocks-2.2.2.tar.gz" -O && tar -xzvf luarocks-2.2.2.tar.gz && cd luarocks-2.2.2 && ./configure && make build && make install'
3. Your error message indicates, that the file you are trying to untar is actually not a tar.gz file despite its name. This can be verified by simply examining the file you just downloaded (e.g. with the file command): It is actually an HTML file with the server response of the curl request, which contains the message of a permanent redirect. In order for curl to follow the server redirect, you have to invoke it with -L.