Compose: docker-compose up creates directories on host system

[strk]

Running docker-compose up results in creation of directories specified via volumes directive on the host system. I think it would be better to error out if the host directory does not exist.

[strk]

besides, the directories are created by the "root" user, even if docker-compose up was called by an unprivileged user (other than being allowed to use the docker sockets)

[dnephin]

The directories are created by docker engine, not by compose itself, so this change would need to be made in the engine.

16072 was the only related issue I could find , but it's for named volumes, not host volumes.

@thaJeztah do you know if there is any talk of this?

[thaJeztah]

Yes, we marked that "feature" as deprecated in 1.9, and it's targeted for removal in 1.11; https://github.com/docker/docker/blob/master/docs/deprecated.md#auto-creating-missing-host-paths-for-bind-mounts

Auto-creating missing host paths for bind mounts

Deprecated in Release: v1.9

Target for Removal in Release: 1.11

When creating a container with a bind-mounted volume-- docker run -v /host/path:/container/path -- docker was automatically creating the /host/path if it didn't already exist.

This auto-creation of the host path is deprecated and docker will error out if the path does not exist.

[dnephin]

Excellent. Closing this issue since it's being fixed by engine.

[dnephin]

Thanks @thaJeztah !

[strk]

Thanks everyone!

[mitar]

Nooo. This was great feature. How can one then make sure paths exist if they do not yet? You have to wrap docker-compose then into a script?

[mitar]

I think this signals well a need for the hooks: https://github.com/docker/compose/issues/1341

[verglor]

Hi @dnephin and @thaJeztah, this was un-deprecated so the problem still exists.
It was solved by a new --mount flag instead of --volume (see #13121).
Is it possible to use this new flag for mounting volumes in docker-comopse ?
Can we reopen this issue, if it is not possible yet ?

[thaJeztah]

@verglor If I'm not mistaken, the "long form" volumes syntax will use the mount API, thus not create local directories (unless the daemon is too old to support that API); see https://github.com/docker/compose/pull/5490

[michaelarnauts]

I'm not sure if there is an existing issue for the mount syntax in docker-compose. There are a lot of issues that specify "mount", but most handle issues with the volumes.

[thaJeztah]

The "volumes" key in the compose-file is used both for (named) "volumes" and for bind-mounts (if you specify a host path instead of a volume name, or, in the long-form when specifying type: bind). Using the long form syntax should use the mount API, which prevents paths on the host from being created by the daemon;

    volumes:
      - type: bind
        source: ./some/path/on/host
        target: /some/path/in/container

[speller]

So, how a volume should be defined in docker-compose.yml file to allow a host directory to be automatically created without any additional custom magic scripts?

[thaJeztah]

@speller the shorthand syntax will do that; https://docs.docker.com/compose/compose-file/#short-syntax-3

volumes:
  - ./some/path/on/host:/some/path/in/container

Be aware though, that:

• the path will be created on the host where the _daemon_ runs (which might be a remote host, or a VM), not on the client. (Docker Desktop makes that largely "transparent", but other setups might not have the same)
• if the path on the host points to a _file_, and that file is missing, that docker will create a _directory_ in that location

Those were the reasons for not auto-creating the paths when using the long-hand syntax, as they've been cause for lots of confusion (and unexpected behavior).

[speller]

@thaJeztah It doesn't in my case. That's why I've found this topic on google. I have a stack deployed with the docker-compose file. I'm using the short syntax. And docker fails to start containers if their volumes' host directories do not exist. The parent directory of the volume directory exists (and have the root:root ownership). But docker can't create a subdir for the volume. If I create the volume directory with sudo (the volume directory has the root:root ownership) container starts successfully and change directory ownership to polkitd:root.

[thaJeztah]

@speller probably better to open a new ticket for that with details (information about the version of compose you're using, as well as engine version (output of docker version and docker info); if possible with minimal steps to reproduce. Are you only seeing that issue when using compose, or also when doing the same with docker run ?

[vaishnoavi]

@thaJeztah : I am also facing the same issue. In my Dockerfile, I am changing the permission of some folder to non-root user.
chown -R non-root-user /usr/local/app_name
When i try to start the container from docker-compose up or docker run, I am mapping the logs folder from the container to the host
-v /path/already/present/this_folder_does_not_exists:/usr/local/app_name/logs.
When i start the container, this_folder_does_not_exists gets created with the root user rather than non-root-user user. Can you provide me some info ?
My docker version : 17.05.0-ce

[volkflo]

@thaJeztah I tried the long volume syntax as you mentioned above, but is there a way to print a warning if the file or directory on the host is missing?

[thaJeztah]

When using the long syntax, it should print an error and fail;

version: "3.7"
services:
  web:
    image: nginx:alpine
    volumes:
      - type: bind
        source: ./some/path/on/host
        target: /some/path/in/container

docker-compose up
Creating network "repro-2781_default" with the default driver
Creating repro-2781_web_1 ... error

ERROR: for repro-2781_web_1  Cannot create container for service web: invalid mount config for type "bind": bind source path does not exist: /Users/sebastiaan/Projects/repro/repro-2781/some/path/on/host

ERROR: for web  Cannot create container for service web: invalid mount config for type "bind": bind source path does not exist: /Users/sebastiaan/Projects/repro/repro-2781/some/path/on/host
ERROR: Encountered errors while bringing up the project.

[volkflo]

@thaJeztah Thanks for the example. My problem was that I tried this with an existing setup.
But it is only working after running docker-compose down -v.
I think this is the correct behavior that is only working for ne created mounts?

[rulatir]

Nooo. This was great feature. How can one then make sure paths exist if they do not yet? You have to wrap docker-compose then into a script?

You have to anyway. Without auto-creation docker will error out. With auto-creation it will create the missing directories as root, which is equally undesirable. There is simply no way to avoid having to supply the missing functionality by writing some code.

[mitar]

With auto-creation it will create the missing directories as root, which is equally undesirable.

There could be a config to configure user under which to create the directory.

[dnck]

With auto-creation it will create the missing directories as root, which is equally undesirable.

There could be a config to configure user under which to create the directory.

That would be great! I'm currently having the issue that docker-compose up creates the directories I need (great), but when I want to tear them down and build them back up again using Jenkins, my build fails because Jenkins can't tear down root owned volumes created by Docker.

[rulatir]

There could be a config to configure user under which to create the directory.

There could be? Optimist!

[qiangli]

There could be a config to configure user under which to create the directory.

or by default under: $USER:$USER

[nicolabeghin]

Any news on this? @thaJeztah tried your suggestion but still not failing if folder missing (maybe due to the compose 3.3 version?) Thanks a lot!

version: "3.3"
services:
  web:
    image: nginx:alpine
    volumes:
      - type: bind
        source: ./some/path/on/host
        target: /some/path/in/container

[thaJeztah]

@nicolabeghin I tried reproducing, but I'm getting the error (as expected);

docker-compose up

Creating network "composetest_default" with the default driver
Creating composetest_web_1 ... error

ERROR: for composetest_web_1  Cannot create container for service web: invalid mount config for type "bind": bind source path does not exist: /Users/sebastiaan/Projects/test/composetest/some/path/on/host

ERROR: for web  Cannot create container for service web: invalid mount config for type "bind": bind source path does not exist: /Users/sebastiaan/Projects/test/composetest/some/path/on/host
ERROR: Encountered errors while bringing up the project.

@nicolabeghin after you ran docker-compose up, was a some/path/on/host directory created in your current directory? Could you also check if a DOCKER_API_VERSION environment variable is set in your environment?

[nicolabeghin]

@thaJeztah thanks a lot for your followup! below the whole thing.

I confirm there's no DOCKER_API_VERSION environment variable set.

I must outline: running from scratch it did led to error as you suggested, but it didn't anymore after trying to change the version from 3.3 to 3.7 in order to check the minimum version required for this. After reverting back to 3.7, as shown above, it didn't fail anymore in any case and with any version.