Compose: Define host volumes in the `volumes` section of the Compose file

[bigfoot90]

It seems that there is not the possibility to mount a host directory using the volumes section using the versrion 2

I would propose an configuration example

volumes:
    data:
        driver: host
        driver_opts:
            source: /host/directory

My use case:
docker-compose.yml

services:
    nginx:
        # ...
        volumes:
            - data:/var/www

    php:
        # ...
        volumes:
            - data:/var/www

    symfony:
        # ...
        volumes:
            - data:/var/www

volumes:
    data:
        driver: host
        driver_opts:
            source: /tmp/data

Override docker-compose.prod.yml

volumes:
    data:
        driver: host
        driver_opts:
            source: /home/prod/data

[dnephin]

host is not a driver type, host volumes are treated differently.

I believe the only way to do this is to inline the path in the service:

nginx:
  ...
  volumes:
    - /tmp/data:/var/www

[bigfoot90]

Thank you for your responce.

host was for example only, you can implement this differently.

Is there any chance for having having such feature in the next release?
It would be good to be able to overwrite only the volume configuration instead of configure different volumes for each service.

[dnephin]

It is possible, but at this time it's not a high priority, we can keep the issue open to track it, and see if there is other interest in this feature.

[nicodmf]

i'm interessing

[x007007007]

i'm interesting

[glorquet]

+1

[tomfotherby]

Did you try using driver_opts in the volumes definition? This worked for me:

services:
  php:
    volumes:
      - app_sourcecode:/var/www

volumes:
  app_sourcecode:
    driver_opts:
      type: none
      device: $PWD
      o: bind

This was with docker-compose 1.11.2 and docker 17.03.0-ce with Ubuntu 16.10 as the host OS. There does seem to be a bug with folder ownership being reset to root though - ref: https://github.com/docker/compose/issues/3270.

[Hasnayeen]

yes love to have this feature

[patuf]

+1
Update: The solution by @tomfotherby seems to work for both compose and docker stacks 17.04.0-ce . The syntax does seem clunky, though.

[jmarcos-cano]

@patuf @bigfoot90 can you point me to a direct example of this feature?

[patuf]

@jmarcos-cano I'm not sure I got your request right - do you want to see how it's working or do you want to know why we might need to have bind-mounted volumes in docker-compose?

[jmarcos-cano]

@patuf I want to see how to use it, I'm not able to find any documentation

[patuf]

@jmarcos-cano Please see the comment from @tomfotherby in this very thread, dated 27 March. Replace $PWD with your preferred host directory (absolute path) and it should work.
If (I doubt it, but mentioning it just in case) you have difficulties understanding tomfotherby's suggestion, you should check the Compose file reference first.
Keep in mind that it doesn't work for bind-mounting single files. It works only for directories. Also, I never tried it on Windows and don't intend to.

[tomfotherby]

Replace $PWD with your preferred host directory (absolute path) and it should work.

For me, on Linux, it works using literally $PWD but yes, if you can hardcode the absolute path that might be more stable because I found some inconsistencies on using either $PWD or ./ or other relative paths depending on whether on Mac or Linux.

[rakshazi]

Does not work for me.
docker-compose version:

docker-compose version 1.13.0, build 1719ceb
docker-py version: 2.2.1
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

and:

docker-compose version 1.16.1, build 6d1ac21

docker-compose.yml:

version: "2"
services:
  update:
    image: composer
    command: ["update", "--no-dev", "--ignore-platform-reqs", "--no-ansi", "--no-interaction", "--no-progress", "--no-scripts", "-a", "-d", "/app"]
    volumes: ['code:/app']

volumes:
  code:
    driver_opts:
      type: none
      device: $PWD
      o: bind

Error:

WARNING: The PWD variable is not set. Defaulting to a blank string.
ERROR: Cannot create container for service update: missing device in volume options

PS: echo $PWD works as expected (returns required path)

[bernhardberger]

has anyone benchmarked the bind mount option? I just tried this on Ubuntu Zesty 17.04 and it seems WAY slower on a read heavy php app (TYPO3 CMS) spread across 2 containers (alpine apache2 and alpine php 7.0)

[kapad]

@tomfotherby 's solution works but the syntax is way to complicated. It took me a while to find this issue and the workaround.

And since it doesn't work on all environments, it's not really usable.

Has anyone found a better solution?

[katcaola]

@rakshazi Did you try making your docker-compose file version 3.2+ rather than 2?

The documentation says "the long syntax is new in v3.2". Or perhaps you might need two dollar signs?

[rakshazi]

No, I missed it, thank you!

[kcjonson]

Was able to mount a named, shared volume with the following config:

volumes:
  shared:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: "${PWD}/shared"
services:
  api:
    build: ./packages/api
    command: npm run start
    volumes:
      - ./packages/api:/usr/packages/api
      - shared:/usr/packages/api/src/shared
  web:
    build: ./packages/web
    command: npm run start
    volumes:
       - ./packages/api:/usr/packages/web
      - shared:/usr/packages/web/src/shared

However, the docs really aren't much of a help at the moment!

[mhyousefi]

I am trying to define global volumes in my docker-compose.yml in order to reuse my volumes in different services. I am not sure how to define it. I am having the following config:

elk:
  container_name: elk
  image: sebp/elk
  ports:
    - "5602:5601"
    - "9200:9200"
    - "5044:5044"
  volumes:
    - logstash_dir:/logstash:ro

filebeat:
  container_name: filebeat
  hostname: filebeat
  image: docker.elastic.co/beats/filebeat:6.3.0
  user: root
  command: ./filebeat -c /filebeat-stuff/filebeat.yml -E name=mybeat
  volumes:
    - filebeat_dir:/filebeat-stuff

volumes:
  logstash_dir:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /path/to/logstash/folder

  filebeat_dir:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /path/to/filebeat/folder

When I run docker-compose up elk, I get the following error:

ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for volumes: 'logstash_dir'

When I run docker-compose up filebeat, I get the following error:

ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for volumes: 'logstash_dir'

[shoeffner]

@mhyousefi Are you using version: "3.2" or higher?

From the docs:

Note: The long syntax is new in v3.2

edit: I just saw that @katcaola already said this as well: https://github.com/docker/compose/issues/2957#issuecomment-394836057

[mhyousefi]

Thanks to a response by @thaJeztah in this issue, I was able to resolve my issue:

"Because you're using the compose-file V1 schema, and that schema does not have support for defining volumes (i.e., no volumes: section); https://docs.docker.com/compose/compose-file/compose-file-v1/#volumes-volume_driver

So when parsing your compose-file with the V1 schema, volumes: is seen as the name of a service, and logstash_dir: is seen as an option for that service, which of course is not valid."

[agowa338]

This does currently not work with files. If one specifies a file, an error message is thrown telling that it is not a folder.

version: '3'
volumes:
  code:
    driver_opts:
      type: none
      device: /etc/nginx/nginx.conf
      o: bind

[alexlenail]

Might someone also be able to point me to where these driver_opts are documented? Not sure what I'm doing with type: none etc..

[agowa338]

@zfrenchee That might look like a trivial task, but the driver_opts are dependent on the used volume driver, in this case the default "local" is used. But that default volume driver is currently not that well documented as it should. There are just many references to it, but no specific documentation regarding itself. Maybe it helps you anyway:

• https://docs.docker.com/compose/compose-file/#driver
• https://docs.docker.com/engine/extend/plugins_volume/#volumedriver
• https://docs.docker.com/storage/storagedriver/
• https://docs.docker.com/storage/storagedriver/select-storage-driver/
• https://docs.docker.com/storage/volumes/#differences-between--v-and---mount-behavior

[thaJeztah]

https://github.com/docker/compose/issues/2957#issuecomment-437291910

But that default volume driver is currently not that well documented as it should

The main reason for that is that those options are not defined by docker, but by what's supported by Linux mount; docker passes those options to mount, so it depends on the host on which the daemon runs, and what's supported there.

• https://docs.docker.com/storage/storagedriver/
• https://docs.docker.com/storage/storagedriver/select-storage-driver/

Note that those are not related to volumes; the storage-drives are the "copy-on-write" filesystems used for storing your container's filesystems, and the image layers.

I had some example / explanation written down somewhere; modified it slightly; perhaps this is useful to understand the "under the hood" mechanisms;

What options can be passed to the "local" driver?

When creating a volume with the "local" (default) driver, and options set, the --opt options are passed on to mount MOUNT(8) (or, more accurately, they're passed to the mount syscall MOUNT(2)).

So to know what options are supported; check those man-pages. Note that some options are dependent on what is supported by the host on which the daemon runs (i.e., if you want to use nfs or other types, those must be supported by the host).

What happens "under the hood"

To get a bit more in-depth (original examples come from https://github.com/moby/moby/issues/19990#issuecomment-248955005);

The example below:

docker volume create --opt type=none --opt o=bind --opt device=/host-path  myvolume

Is equivalent to the following in the compose file:

version: '3'
volumes:
  myvolume:
    driver_opts:
      type: none
      device: /host-path 
      o: bind

And roughly translates to the following being done by the docker daemon (don't run the commands below as they mess with files/directories inside /var/lib/docker, which you should never do!);

1. Receive an API call, and store the volume definition (reserve the name and so on)

2. Create the volume storage location for the volume:

   bash mkdir -p /var/lib/docker/volumes/myvolume/_data/

3. Call mount with the given options;

   bash mount -t none -o bind /host-path /var/lib/docker/volumes/myvolume/_data/

And, when _starting_ a container that uses the volume;

docker run -v myvolume:/container-path -dit --name mycontainer  busybox:latest

Docker mounts that volume (which itself is a mount) into the container; something lke:

mount -t none -o bind /var/lib/docker/volumes/myvolume/_data/ /var/lib/docker/overlay2/0b223...etc..../container-path

An example

On a Linux machine you could try this (but you may have to cleanup afterwards :sweat_smile:)

Set up a directory to play with;

mkdir volume-example
cd volume-example

Create the "host" content:

mkdir -p host-path/with/subdirectories
echo "I haz content" > host-path/hello-world

Steps below are an approximation of what the daemon does when creating the volume;

Create the "volume storage location";

mkdir -p var-lib-docker/volumes/myvolume/_data

Mount host-path onto the volume storage path;

mount -t none -o bind host-path var-lib-docker/volumes/myvolume/_data

And when starting the container;

mkdir -p var-lib-docker/overlay2/some-long-id/container-path
mount -t none -o bind var-lib-docker/volumes/myvolume/_data var-lib-docker/overlay2/some-long-id/container-path

Et-voilå, this is roughly what it looks like when the container is running :tada:

tree
.
|-- host-path
|   |-- hello-world
|   `-- with
|       `-- subdirectories
`-- var-lib-docker
    |-- overlay2
    |   `-- some-long-id
    |       `-- container-path
    |           |-- hello-world
    |           `-- with
    |               `-- subdirectories
    `-- volumes
        `-- myvolume
            `-- _data
                |-- hello-world
                `-- with
                    `-- subdirectories

14 directories, 3 files

In the output above, you see the hello-world file and with/subdirectories appear three times;

• the host-path/.... files are the actual content ("on the host")
• the var-lib-docker/volumes... is the storage for the "named volume", and in this case, is not a _copy_, but a _mount_ of the "files on the host"
• the var-lib-docker/overlay2... is where the "named volume" is mounted in the container

Hope this helps :+1:

[agowa338]

@thaJeztah Can you please also add that to the official docs?

[alexlenail]

Thanks @thaJeztah !
One other question: what is the proper way to use a 'local' volume with a remote docker-machine setup in docker-compose v3? Previously, docker-machine was doing some magic to copy local files (e.g. app source code) from my local environment to my remote docker machine. That doesn't seem to be the case in v3 anymore.

Previously I had:

version: '2'
services:
  nginx:
    image: nginx:latest
    ports:
      - 80:80
    volumes:
      - /Users/alex/app/web/static:/usr/src/app/static

which copied the contents of my app/web/static to /usr/src/app/static each time I docker-compose up'd

Now I have

version: '3'
services:
  nginx:
    image: nginx:latest
    ports:
      - 80:80
    volumes:
      - static:/usr/src/app/static

volumes:
  static:
    driver: local
    driver_opts:
      type: none
      device: /Users/alex/app/web/static
      o: bind

which works locally, but not in docker-machine env remote. Is the solution to use rsync as this suggests?

Perhaps this is better suited as a stackoverflow question, but I suppose it can be though of here as a request for more documentation about volumes in circumstances like there.

[dbjpanda]

@thaJeztah That is a very clear description of o:bind . Can you please give me some pointers how can I enable acl on those bind mounted named volume ? I tried to pass o: bind, acl but it didn't help.

[tim-pl-m]

i'm interessing

i'm also "interessing" xD

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[agowa338]

not stale

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.

[TidalWave123]

How do I mount a volume that doesn't copy into the host, but passes the data through to the FTP share?

If I use the bind command when using the volumes in docker-compose I can't get it to work. I'm willing to invest the time to figure this out, but first I would like to know if the bind option would solve my problem. I'm currently trying to mount a mounted FTP share.

So I have an FTP shared mounted on my host. It's an external drive offsite with lots of storage space. Then I run docker on my host and I try to write to the FTP share, but it also creates an "overlay" of data on top of my host as well as writing to the FTP share. This is terrible since my host only has 25 Gigs of freespace, while the FTP share has over a TB.

I'm running ownCloud and want to write directly to the mount point without the copy of data that is existing in the container. Is there anything I can do? my syntanx is a bit different than the documentation so I'm having an impossible time using the bind command anyways.

Here is the syntax:

services:
version: '2.1'

volumes:
  files:
    driver: local
  mysql:
    driver: local
  backup:
    driver: local
  redis:
    driver: local
  owncloud:
    image: owncloud/server:${OWNCLOUD_VERSION}
    restart: always
    ports:
      - 10.0.8.1:${HTTP_PORT}:8080
    depends_on:
      - db
      - redis
    environment:
      - OWNCLOUD_DOMAIN=${OWNCLOUD_DOMAIN}
      - OWNCLOUD_DB_TYPE=mysql
      - OWNCLOUD_DB_NAME=owncloud
      - OWNCLOUD_DB_USERNAME=owncloud
      - OWNCLOUD_DB_PASSWORD=owncloud
      - OWNCLOUD_DB_HOST=db
      - OWNCLOUD_ADMIN_USERNAME=${ADMIN_USERNAME}
      - OWNCLOUD_ADMIN_PASSWORD=${ADMIN_PASSWORD}
      - OWNCLOUD_MYSQL_UTF8MB4=true
      - OWNCLOUD_REDIS_ENABLED=true
      - OWNCLOUD_REDIS_HOST=redis
    healthcheck:
      test: ["CMD", "/usr/bin/healthcheck"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
#     - /mnt/ftp_share/ownCLOUD:/mnt/data
#      - type: bind
      - files:/mnt/data
#     - type: bind
#        source: /mnt/ftp_share/ownCLOUD
#        target: /mnt/data/files/admin/files/SMBstorage
      - /mnt/ftp_share/ownCLOUD:/mnt/data/files/admin/files/SMBstorage
  db:
    image: webhippie/mariadb:latest
    restart: always
    environment:
      - MARIADB_USERNAME=owncloud
      - MARIADB_PASSWORD=owncloud
      - MARIADB_DATABASE=owncloud
      - MARIADB_MAX_ALLOWED_PACKET=128M
      - MARIADB_INNODB_LOG_FILE_SIZE=64M
    healthcheck:
      test: ["CMD", "/usr/bin/healthcheck"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - mysql:/var/lib/mysql
      - backup:/var/lib/backup

  redis:
    image: webhippie/redis:latest
    restart: always
    environment:
      - REDIS_DATABASES=1
    healthcheck:
      test: ["CMD", "/usr/bin/healthcheck"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - redis:/var/lib/redis

#end

It took me ages to even get docker to write to the FTP share. I had to mount it with all kinds of flags. So I mount the ftp share inside my Ubuntu 18 host. Via a cifs, command, come to think of it, my FTP share might be an SMB share, but anyways, it doesn't matter the method I used to mount to the host, because docker can write to the mount point just fine. But docker creates this overlay of data on top of the host. So every file I upload to the SMB share, goes into my 1TB external drive, but also gets copied into the volume so the host loses freespace.

Is there anything that can be done about this wasted space?

[sbezugliy]

Should be useful for cases when volume already defined at Dockerfile and no possibility to redefine it at docker-compose.yml as directory binding.

This didn't solve this case: https://github.com/docker/compose/issues/2957#issuecomment-437487755

Tried it just now with postgres:12.2-alpine and have no success using described above and methods as with previous versions. Data still saves at volume, not at host directory. I see only way - to rewrite Dockerfile.

[u0m3]

I have been using this http://blog.code4hire.com/2018/06/define-named-volume-with-host-mount-in-the-docker-compose-file/ for a while. No ideea if it covers all the use cases described here.

[davetapley]

I just wanted to point out that I too suffered from some confusion around this issue, which I summarized in https://github.com/docker/compose/issues/7524#issuecomment-642821798.

See also: https://github.com/docker/compose/issues/6697

[vasilvestre]

Is there a way to run it inside wsl 2 so I have no file path problem from my windows executing jetbrains ide ?
The actual problem is that you need absolute path but that wsl generate it and as it have different path, ask to recreate the named volume all the time even if the docker volumes run inside wsl linux filesystem.
I know windows will soon allow GUI to run inside WSL but I would need a workaround.

[agowa338]

try prefixing the path with /mnt/c/ if your docker runs using wsl2 that could work.