Compose: docker-compose terminates if build directory is missing

[hholst80]

Summary:

docker-compose seems to pre-check that all build directories exists before running any commands. Even for commands where the build directory is not needed, like docker-compose ps or docker-compose run ...

Command:

docker-compose ps

Result:

ERROR: build path /omnicoder either does not exist, is not accessible, or is not a valid URL.

Expected result:

        Name                       Command               State              Ports
---------------------------------------------------------------------------------------------
api-server_1   npm start                        Up      3000/tcp
cron_1         /usr/sbin/crond -f -d 8 -c ...   Up
disbatcher_1   /usr/bin/tini -- python -m ...   Up
mongodb_1      /entrypoint.sh mongod            Up      27017/tcp
nginx_1        nginx -g daemon off;             Up      0.0.0.0:443->443/tcp, 80/tcp

[trodrigues]

Just ran into this issue myself, in a situation where someone was trying to run a service and getting an error about a directory not existing.

The file in question defines multiple services. One of these services can run just this application (a REST API) and another one can run this application, and also build and run an image for an UI which talks to this API.

The person in question wanted only the API so they ran the service which starts only the API. But because there is a definition for that other service which runs both API and UI in the same file, and because this person didn't have the directory where the UI is built from, they got an error, preventing them from running their desired service.

[jhagege]

+1

[CodeCorrupt]

I'm currently running into this issue with Jenkins deployment. I use the docker-compos build command to build the images locally, then push them to the registry. From there I go to the production server and want to dockere-compose rm -sf && docker-compose up copying over only the compose file and I get this error.

I'm getting around the issue by simply creating an empty directory. but it's a rather clunky work around.

[afeilulu]

+1

[hedleyroos]

Docker compose is very useful for non-developers to quickly get an env up and running (eg. to demo) but this issue means you have to distribute a few files and not just a docker-compose.yml. It would be a very useful fix.

[italktocomputers]

I can confirm this issue is present in docker-compose version 1.23.2, build 1110ad0.

[yurii-piets]

I'm running into this same issue.

[loeffel-io]

same here

[ariebrainware]

Any update?

[cbrunnkvist]

Same.

I'm getting around the issue by simply creating an empty directory. but it's a rather clunky work around.

Some of my (monorepo-) services have their build context outside (../xxx/yyy) of where their compose files live, so this make any such workaround even more clunky for me currently.

I suppose that the "correct" solution is to just not rely on _compose_ but on _swarm_ mode for all deployment since the build section is ignored there altogether probably for this reason.

[SpecialForce3331]

Version 1.24 problem still here!

[cbrunnkvist]

The best approach is to _split_ files and provide docker-compose with multiple -f stanzas, for example maybe the bare necessities to run a service can remain in the docker-compose.yml file, but you can add in the build: part (the yaml structs will be merged) only during build operations by using e.g. docker-compose -f docker-compose.yml -f build.docker-compose.yml build for your project.

Taking this into account - terminating on missing build dir (which could potentially result in mis-built images, which would be more difficult to track downstream) could be considered a feature rather than a bug.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hholst80]

Same.

I'm getting around the issue by simply creating an empty directory. but it's a rather clunky work around.

Some of my (monorepo-) services have their build context outside (../xxx/yyy) of where their compose files live, so this make any such workaround even more clunky for me currently.

I suppose that the "correct" solution is to just not rely on _compose_ but on _swarm_ mode for all deployment since the build section is ignored there altogether probably for this reason.

That's exactly why you should not use Swarm. Don't let Docker force to use Swarm just because they won't fix compose. Make a vendor patch set like mine, or you can use multiple compose files like suggested. That is what I am using now.

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.

[arendarenko]

v1.25.4 still no progress

[openface]

I had hoped that specifying --no-build to the up command would somehow avoid this behavior. Sadly, it does not.

[vcaboara]

docker-compose -v
docker-compose version 1.26.2, build eefe0d31
docker-compose rm -sf
ERROR: build path /whatever either does not exist, is not accessible, or is not a valid URL.

[cbrunnkvist]

After some consideration, I still think it is actually a good thing that it just fails even when given --no-build _because it affects all sub-commands_. If it depended on "no build" then all other commands would need to implement a "no build" argument as well - docker-compose config for example, which only validates and canonicalizes the config.

Once again, the solution for a sometimes missing build dir is is to _break apart your composition_ into multiple pieces and let them cascade together, using either multiple -f parameters or (whichever you prefer) :-path separator in the COMPOSE_FILE variable.

Practical example:

$ cat docker-compose.build.yml 
version: "3.2"
services:
  app:
    build: ../src/app

$ cat docker-compose.deploy.yml 
version: "3.2"
services:
  app:
    image: registry.local/my/app:latest

$ docker-compose -f docker-compose.build.yml -f docker-compose.deploy.yml config
services:
  app:
    build:
      context: /Users/me/checkout/src/app
    image: registry.local/my/app:latest
version: '3.2'

$ COMPOSE_FILE="docker-compose.build.yml:docker-compose.deploy.yml" docker-compose config
# same as above

[scranen]

I don't think that your example is the situation that the original poster was reporting.

Impractical example:

$ cat docker-compose.yml
version: "3.2"
services:
  foo:
    build: src/foo
  bar:
    build: src/bar

$ docker-compose build foo
ERROR: build path src/bar either does not exist, is not accessible, or is not a valid URL.

$ why do you care?!
zsh: no matches found: care?!

If you are using docker-compose in 99% of cases to set up an environment with app1 and app2, then it is very nice to be able to specify that entire environment in one docker-compose file. I would be very unhappy if in my build setup I had to pass 5 -f stanzas every time I want to test something, in that case I would end up writing a layer of tooling around docker-compose, which is what I am trying to avoid. So for now, I'm also using the trick of just creating the build directories.

Note that the check for having a build directory is a strange one to do at this point, because there is absolutely no guarantee that you can build an image out of it. In particular, the check that there is a Dockerfile is done only when a build is requested.

$ mkdir src/bar
$ docker-compose build bar
ERROR: Cannot locate specified Dockerfile: Dockerfile
$ docker-compose build foo
Building foo
... why is it now safe to build foo, while it wasn't before?

I think what the original poster and the people who were chiming in before are asking, is that the check for the existence of the build directory is done at the same place as the check for the existence of a Dockerfile. This would be more consistent, and allows people to keep using docker-compose in the lightweight fashion they were used to.