Compose: Named Volumes & Persistance in version 3

[rivaros]

This is basically a question about named volumes.
Starting from version 3 file format docker-compose does not allow volumes-from which was deprecated in favor of named volumes.

But there is a problem - with named volumes you cannot bind to local host folder & make it persistent.
So, for example, if I want to share my project folder among nginx, fpm and mysql docker containers, I need to specify a volume mapping setting for each of them.

It's not possible to just create one named volume mapped to project folder & then simply reuse it in all containers.

Why this is not possible? I am aware of the third-party docker plugin like https://github.com/CWSpear/local-persist but as all third-party things, this does not make me feel I should use it (is it stable or maybe it is slow). Are there any particular reasons (drawbacks) why it was not implemented in the core?

[shin-]

One solution would be to declare each bind individually, like:

version: '3'
services:
  mysql:
    volumes:
      - ./project:/opt/project
    ...
  nginx:
    volumes:
      - ./project:/opt/project
    ...
  fpm:
    volumes:
      - ./project:/opt/project
    ...

This is more explicit than using volumes_from, which might actually be a good thing.

On the other hand, why do your nginx and mysql services need access to your code?

[rivaros]

@shin- the solution you describe is what I use now (and it seems not good for me, because of duplication & increased typoo possibility), but since V3 deprecated volumes-from, it means that docker team decided it's wrong approach.

There are scenarios where it is desirable to share one volume among multiple containers. For example nginx - to serve static content from project folder, fpm - to access both dynamic and static

Docker now says "use named volumes", but omits "persistence". Why is it?

If we don't use one volume per multiple containers we also have another issue - increased number of orphaned volumes after several successful/unsuccessful builds.

p.s.: also aware about docker system prune but that's not what I'd like to spend time every day

[markvillar]

@rivaros I have the same problem. I also want to use named volume, but there seems to be no way to tell docker where to mount the volume from the host?

[masterful]

I'm having difficulty with this as well. With the updates to docker-compose in v3.2 and long-syntax there doesn't appear to be a way to have names/aliases for bind mounted volumes. It appears that the source is used for the name for volume mounted volumes, but the local path for bind mounted volumes...

With the removal of volumes_from support, I was wondering if it might be possible to define named volume support for bind mounted volumes, which would allow for single local path definition that can be shared between several services...

version: '3'
services:
  web:
    ...
    volumes:
      - common_code
  worker:
    ...
    volumes:
      - common_code
volumes:
  - type: bind
    name: common_code
    source: /local/path
    target: /application/path

It was mentioned that local-persist could be used as an approach to named local volumes, but I echo the question of why this isn't present in core. However, an approach like that would allow for even more nuanced behaviour like:

version: '3'
services:
  worker:
    ...
    volumes:
      - common_code:/application/path
  different-worker:
    ...
    volumes:
      - common_code:/different/application/path
volumes:
  - type: bind
    name: common_code
    source: /local/path

Is there a reason we couldn't have named/aliased bind-type volume definitions inside the docker compose file?

[rivaros]

Upvoting this again. Any comments from off team why it's not implemented?

[hmhofman]

Like @rivaros said in comment 299299853. Any comments?

We use docker configuration inside a sub folder of a git repository.
This means we have nginx/php machines pointing to ../ and ../public/
We should be able to set that source folder in named volumes.

Then again, I'm not even seeing volumes mounted on the way @shin- described in comment 289867611

[GodLesZ]

Months ago and still no answer from docker team? 😞

[nicodmf]

Hello,

Thanks for docker, but i agree with others, the volume from has no replacement now.

Applications which i use are in the same states that persons spoken before, we have :

• a app server container
• a command container
• a www server container

each of there works on the same data (assets, appfile, etc...). Traditionnal way to deploy, capistrano and same tools, don't works as they use many symbolic links not mounted by docker. The only way to deploy cleanly is consequently to use an image with data only, purge the volume on host and up the containers.

Another solution could be "use one container with all inside" : web server, app server, all cli command, but in this case why use docker ?

I think, the volume from or shared app data (code, assets aren't data, just the result of the build) should be reevaluated, maybe with just an option to volume, by example :

#...
  volumes:
  - type: volume
    source: mydata
    target: /data
    volume:
         copy_on: start # or up or empty or false
#...

Thanks for docker and docker-compose.

[aaronlifton]

👍

[reypm]

Wowwwww!!! is almost 2018 and none has been assigned to this? This is a "must" since I am running into the same issue and there is a few workaround in this topic and out there but why haven't as part of the core? Upvoting this

[SimonLammer]

How is this still not being implemented? At least I'd like to know an official workaround for this problem!

[luislobo]

+100

[robertcsakany]

+1

[hosamshahin]

+1

[rijnhard]

Damn frustrating when using it for local development in complex orchestrations.

[mohanzeal]

+1

[tatyanamincheva]

+1

[JanGolle]

up vote!

[bvisonl]

+1

[fcoedno]

+1

[eduardichim]

Anyone who wants to define it in the top level volumes try this:

#this only works with folders doesn't work with a file
#driver_opts(local)->device: either ${PWD} OR container:/path (production if you have a data container)
volumes:
  data_app:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}
      o: bind
  data_web:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/docker_files/nginx_conf
      o: bind

You don't need a plugin and now you can use in your service something like this:

web:
    image: nginx:1.10 # from an image
    volumes: # mount path of the volume
      - data_app:/var/www
      - data_web:/etc/nginx/conf.d
    networks: # internal network for accessing port from other containers
      - backend
    ports: # expose port to be accessed from outside
      - 8080:80

When I try the example above

version: '3'
services:
  worker:
    ...
    volumes:
      - common_code:/application/path
  different-worker:
    ...
    volumes:
      - common_code:/different/application/path
volumes:
  - type: bind
    name: common_code
    source: /local/path

I get
```
In file './docker-compose.yaml', volume must be a mapping, not an array.
````

[dasmedium]

Up-vote. Sounds like more dependency.

[bobba12]

+1

[oxie]

Guys with all the growth of docker and the popularity around it, also enforcing new standards in the field. Please, you have to think of an official and decent way to store data with docker.

[eexit]

In file './docker-compose.yaml', volume must be a mapping, not an array.

@eduardichim Use version >= 3.2.

[rgj8]

+1

[mstraughan86]

+1

[Flyrell]

+1

[gjelenc]

+1

[KevinDanikowski]

+1

[shin-]

+1s just serve to aggravate people subscribed to the issue. If you've got nothing constructive to add, just use the thumbs up button.

[shin-]

This issue can now easily be solved using extension fields, like so:

version: '3.5'
x-project-mount:
  &project-mount
  type: bind
  source: ./project
  target: /opt/project

services:
  mysql:
    image: mysql
    volumes: 
      - *project-mount

  web:
    image: django
    volumes: 
     - *project-mount
     - type: tmpfs
       target: /opt/web-tmpdata

  fpm:
    image: bitnami/php-fpm
    volumes:
      - *project-mount

As a result, I am considering this to be resolved.

[grisaitis]

@shin- does your solution also enable varying target paths in service definitions?

See @masterful's example, last code block: https://github.com/docker/compose/issues/4675#issuecomment-295347797.

[shin-]

@grisaitis Yes, you can use the YAML merge type to achieve that, e.g.

version: '3.5'
x-project-mount:
  &project-mount
  type: bind
  source: ./project
  target: /opt/project

services:
  mysql:
    image: mysql
    volumes: 
      - <<: *project-mount
        target: /opt/target-override

[harpro]

Is there a solution to this problem?

[lddsb]

got ERROR: In file './docker-compose.yml', volume must be a mapping, not an array. too.

[laurencefass]

subscribing.

[WolfgangFahl]

What? This is closed? Instead of a simple syntax supporting name there is now an awkward work-around that is considered the solution?

There are already enough quirks in docker. I personally do not need another one. I'd appreciate this to be reopened and properly fixed.

[KrazyMax]

I totally agree with @WolfgangFahl , I do not understand why such a simple request/need from so many users has not been really taken into account. Since I am a Docker user I have been struggling with this notion of named volume (and other things) and especially how it is not possible to define it once for all at one place instead of everywhere. Then I found this ticket and realized I was not alone looking for any logic here. Then I found myself disappointed in the solution provided. It's a little thing which complexifies Docker one more time for a reason not obvious at all... as if Docker HAD to look complex or so to be considered as a serious business. Damn this is not mandatory, please keep it simple! :-(

[Pacolu]

I am not sure, if I understand your point. So this ticket is about: "What is the syntax to share named volumes with the host?", or?
If so, the answer of @eduardichim is fine and in use in my company for a year and a half or so. So let me give you another example of it:
Depending on your setup, you might have a "default" compose file with named volumes and services.
In addition you might have a docker-compose.production.yml where you persist those volumes, which should be persisted in a productive environment, like databases.
You also might have a docker-compose.local/development/override.yml where you persist everything. The reason why I dont have only one mega file is the readability and the CI, since the CI does not need to persist anything.

So starting the example files:

version: '3.5'

volumes:
  appSource:
  dynamicData:
  database:

services:
  app:
    image: registry.test.de/docker/app
    volumes:
      - appSource:/var/www/html
      - dynamicData:/var/app
    networks:
      - backend
  nginx:
    image: registry.test.de/docker/nginx
    depends_on: ["app"]
    volumes:
      - appSource:/var/www/html
      - dynamicData:/var/app
    links:
      - phpfpm
    networks:
      - backend
  phpfpm:
    image: registry.test.de/docker/php
    depends_on: ["app"]
    volumes:
      - appSource:/var/www/html
      - dynamicData:/var/app
    networks:
      - backend
  db:
    image: mysql
    volumes:
      - database:/var/lib/mysql
    networks:
      - backend
  cron:
    image: registry.test.de/docker/cron
    depends_on: ["app"]
    volumes:
      - appSource:/var/www/html
      - dynamicData:/var/app
    networks:
      - backend
networks:
  backend:
    driver: "bridge"

Now the production YAML

version: '3.5'

volumes:
  database:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/mnt/database # $PWD is linux based and can be changed if necessary to an absolute path
      o: bind

services:
# Possible other stuff

And finally your local environment YAML

version: '3.5'

volumes:
  # $PWD is only available on linux based systems (Linux, MacOs), Windows might need absolute paths
  appSource:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/mnt/src # abs. path ONLY
      o: bind
  dynamicData:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/mnt/dynamicData # abs. path ONLY
      o: bind
  database:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/mnt/database # abs. path ONLY
      o: bind

services:
# Some kind of devbox

With this your files are stored twice. In Ubuntu for example:
/PROJECT_PATH/mnt/src/file
/var/lib/docker/volumes/DIRECTORY_appSource/_data/file

Some notes about this solution:

• With MacOs the performance is bad. About 7-10 times worse than ubuntu. Searching for a solution with mutagen.io
• the directories must exist before the service started up
• other than that this works more or less exactly so on Linux/Mac/Win

So whats the painpoint here, or what do you think should work differently with a "more?" native compose solution?

[infostreams]

For future googlers, the apparently preferred way to share multiple volume definitions between different containers is now this:

# see https://github.com/docker/compose/issues/4675#issuecomment-584526950
x-volumes:
  &volumes
  volumes:
    - type: volume
      source: code
      target: /var/www/html/
    - type: volume
      source: uploads
      target: /var/www/html/storage/app/public/
    - type: volume
      source: unclaimed_uploads
      target: /var/www/html/storage/app/uploads/
    - type: bind
      source: ./production.env
      target: /var/www/html/.env
      read_only: true

services:

  nginx:
    image: nginx:1.17.8
    networks:
      - docker
      - web
    restart: always
    <<: *volumes

  php:
    image: php:7.3-fpm-alpine
    restart: always
    networks:
      - docker
    <<: *volumes

volumes:

  code:
  uploads:
  unclaimed_uploads:

Here both the 'nginx' and the 'php' container will receive the same set of volumes.