Devilbox: Using Docker for Windows to volume-mount a Windows drive into a Linux container is bad

[loxy]

Environment:

• OS: Windows 10 Pro
• docker version: 17.09.1-ce
• docker-compose version: 1.17.1, build 6d101fb0
• Devilbox: v0.12

When you're using Docker for Windows to volume-mount a Windows drive into a Linux container, that volume is done using a CIFS/Samba network share from the Windows host. For lots of reasons, it's highly unlikely that for example Linux Postgres will work correctly when trying to write data to a filesystem backed by NTFS shared with Samba.

So this is not working on Windows:

services:
  pgsql:
    volumes:
      # Mount PostgreSQL Data directory
      - ${HOST_PATH_PGSQL_DATADIR}/${PGSQL_SERVER}:/var/lib/postgresql/data/pgdata

You are getting:

waiting for server to start....FATAL:  data directory "/var/lib/postgresql/data/pgdata" has wrong ownership
HINT:  The server must be started by the user that owns the data directory.
 stopped waiting
pg_ctl: could not start server

Instead, use a persistent (but local to the Linux VM) named volume as detailed here: https://forums.docker.com/t/trying-to-get-postgres-to-work-on-persistent-windows-mount-two-issues/12456/5?u=friism

To summarize: The workaround is to create a (local) volume with:

$ docker volume create --name data-postgresql --driver local

And the docker-compose.yml looks something like that:

services:
  pgsql:
    volumes:
      - data-postgresql:/var/lib/postgresql

volumes:
  data-postgresql:
    external: true

The same is true for MongoDB like this issue states: #160
With this workaround everything works!

I'm not a docker expert, but maybe there's another solution instead of changing the docker-compose.yml file, which is not recommended: ## -- DO NOT EDIT THIS FILE --

[cytopia]

@loxy after having created the local volumes, where on the local filesystem would you access them?

[loxy]

More infos on that:

First of all: bind mounts work in Windows, but not completely. The problem arises for example when dealing with different users and permissions. Postgres runs under a different user than root, for security reasons. This is mandatory. So, while initializing the image it chowns the PGDATA directory. And this breaks somehow the CIFS/Samba network share. This is the reason why not all containers are affected.

Look here:

• https://github.com/docker/for-win/issues/445
• https://github.com/docker/for-win/issues/39

I didn't found a solution for this problem. In Windows, Volumes are the only way to go, for now.

By the way, using volumes is the preferred mechanism for persisting data generated by and used by Docker containers. See: https://docs.docker.com/engine/admin/volumes/volumes/

But you are right, I have no direct access to those data (local volumes in Windows) because it resides in the virtual machine windows uses to establish the Docker environment. To be more precise: Docker for Windows uses a VHDX volume, normally located at C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\MobyLinuxVM.vhdx. I think the data is there, but not directly accessible. I even tried this one:
https://blog.jongallant.com/2017/11/ssh-into-docker-vm-windows/
But no chance...

For Postgres this is OK, because the important thing is, that the data is not lost on removing the container. And no one can holding me back from dumping and restoring the db to a another directory, which works normally.

Moreover, volumes didn't have to be local. They can be located anywhere, i.e. on AWS.

[cytopia]

@loxy I spent some time regarding your request using named volumes.

The problem I have encountered is, that changing versions of a specific service (e.g. mysql) will not actually change their named volumes.

I setup named volumes in docker-compose.yml which are different for each version of mysql, however when actually changing .env and specifying a different mysql version. The new volume is not picked up and it keeps the existing volume of the old mysql version.

Service mysql is using volume "/var/lib/mysql" from previous container. Host mapping "mariadb-10.1" has no effect. Remove the existing containers (with docker-compose rm mysql) to use the host volume mapping.

[cytopia]

@loxy I finally came up with a solution that works with different version. PR is available and awaits testing. Please try it out and let me know how it works for you:

https://github.com/cytopia/devilbox/pull/383

[loxy]

Will have a look!

[cytopia]

@loxy did you have time to check if this resolves your issue?

[loxy]

Sorry for the delay @cytopia.
Everything works out of the box. Great! Thank you!!!

[cytopia]

Ok thanks. Just to double check: you've tested that on "Docker for Windows", not DockerToolbox on Windows?

[loxy]

Yes, "Docker for Windows"

[cytopia]

OK. I will lock this conversation then to prevent any other comments.

Devilbox with Docker volumes

Pure Docker volumes branch is available and should be used from now on. Visit the following link and follow the steps described in the PR

https://github.com/cytopia/devilbox/pull/383

Please add any other arising issue in that PR.

[cytopia]

Devilbox release v1.0.0 (pre-release)

The v1.0.0 Release branch is out and should be used: https://github.com/cytopia/devilbox/pull/416

It is made sure that this branch is stable and will only receive other stable features. It is recommended to use this one (also includes Docker volume feature).

I will close this issue, please add any problems to the above linked PR