Machine: Shared folders should be configurable for local VM drivers

[bfirsh]

[nathanleclaire]

I'm curious how you envision the future of this shaping up. A machine share command or something more like the auto-mount of /Users that we have in boot2docker today?

I've been thinking a little about this and since we have been hitting huge performance issues with VBox shared folders supporting at least a minimal NFS driver (possibly SSHFS / samba too ala the spirit of https://github.com/boot2docker/boot2docker-cli/pull/247) would be a good direction. Obligatory link to Mitchell's article which is a good treatise on why this would be useful. Short story: NFS could probably give us ~30 times better read performance than shared folders (and mostly read performance matters for our needs).

I'm in favor of something like machine share . any time that you want to do something like docker run -d -P -v $(pwd):/code nathanleclaire/webapp, since it doesn't obfuscate the underlying operation much.

[dreampuf]

+1

[Maxim-Filimonov]

:+1: on NFS share. We have experienced very bad performance with shared folder even with vmware fusion. The only tricky part with we have found is that NFS sometimes doesn't bust a cache properly -> need to change file again for machine to pickup changes.

[rawkode]

rsync or nfs would be awesome :+1:

[joshfinnie]

Cannot upvote this enough... would LOVE to see this added to machine! :+1:

[Compufreak345]

+1

[rafabene]

+1

[zymplsi]

+1

[EpocSquadron]

+1 This is the only thing holding me back from rolling toolbox out as a development environment solution for our developers. I have a hacked-together setup similar to docker toolbox that utilizes nfs specifically because vboxfs gives permissions and performance issues.

[fesor]

@EpocSquadron you can try https://github.com/codekitchen/dinghy

[krasi-georgiev]

+1

[Besedin86]

:+1:

[posita]

I just filed Parallels/docker-machine-parallels#29, but I believe Boot2Docker is able to support mounting NFS volumes by itself, which suggests enabling such volumes could be handled in a driver-agnostic way (see, e.g., https://github.com/boot2docker/boot2docker/issues/581#issuecomment-74535277). I'm imagining something like:

% docker-machine create --help
...
  --share-type "driver"     "driver" or "nfs[:<mount_options>]" where
                            <mount_options> are optional NFS options,
                            such as "mapall=501:501,nolock,sync"
...

Then you could do something like:

% docker-machine create --driver ... --share-type "nfs:mapall=$( id -u ):$( id -g )" ...

root or sudo access might be needed on the host to modify /etc/exports (this is how Vagrant works). Alternately, instructions on modifying /etc/exports could be printed for the user as part of the create and start commands (with corresponding removal instructions printed out as part of the rm and stop commands). Another alternative might be an analogy to the docker env command (writting eval-utable instructions to modify /etc/exports accordingly).

[posita]

From @legal90 via this https://github.com/Parallels/docker-machine-parallels/issues/29#issuecomment-164727243 related to my above proposal:

... [T]his is a good idea. ... NFS support should be implemented in both of Boot2Docker and Docker Machine itself, but not in a driver because it is not related to the virtualization system directly.

The good example is Vagrant and how NFS support is implemented there. Following this way, Docker Machine should manage NFS exports on the host side and also it should mount NFS folders on the guest side (during the provision step).
So, this feature is mostly host-specific (Windows doesn't support it) rather that provider-specific (it should work for VirtualBox and VMware by the same way).

[johnsudaar]

+1

[jedipunkz]

+1

[brandontamm]

+1 - If /Users is mounted by default and is larger than my virtual machine's disk size, wouldn't this cause problems?

I am wondering because I use github.com/adlogix/docker-machine-nfs to share large projects like a Magento project. I have a work machine and a personal laptop - the work machine has virtually nothing in /Users and I am able to mount the NFS, but the mount always fails when I try to mount to NFS using my laptop (with large /Users directory)

When I use github.com/codekitchen/dinghy I do not have this problem, because it is mounting only the volume specified in docker-compose.

@nathanleclaire IMO, the file sharing and volumes and mounting is a HUGE blocker for docker (no pun intended) and we should have a choice of how we mount volumes and what is actually mounted. Otherwise we have to use these workarounds to develop projects with giant file structures. For instance, I can't docker-compose up a development Magento project with 30,000 files because the volumes aren't available in time and cause everything to break. So we have to try these side projects for NFS...

Then again, I am new at this but I see SO MANY issues resulting from file shares..

[SashaOv]

+1

[rawkode]

If it helps anybody, I'm using the new --virtualbox-no-share flag (docker-machine) and this script to rsync files:

https://github.com/rawkode/symfony-distribution/blob/master/bin/osx-sync

We should be able to do docker-machine create -d virtualbox --share 'rsync:.:/var/www' development - whatever the syntax ends up - surely this isn't too far away now? :)

[nathanleclaire]

I am wondering because I use github.com/adlogix/docker-machine-nfs to share large projects like a Magento project. I have a work machine and a personal laptop - the work machine has virtually nothing in /Users and I am able to mount the NFS, but the mount always fails when I try to mount to NFS using my laptop (with large /Users directory)

Does it give you an error message specifically indicating that this is the problem?

Have you tried doing the NFS mount manually?

[stkiss]

+1

[jmreicha]

Has there been any movement on this?

It would be great to see something like --share-type with options for NFS, sshfs, rsync, samba, etc. and (maybe?) --share-path src:dest implemented.

[mrmachine]

I use VMware Fusion (not VirtualBox) which has much more performant file sharing. However, I still need to share directories other than /Users to the docker host (boot2docker created with docker-machine). I want to run some docker containers that have access to files on my and other drives physically attached to my iMac. Is there any other workaround? Should I be trying to configure boot2docker to mount volumes from my NAS and iMac into the docker host? Or should I be trying to configure my containers to access the NAS and iMac volumes directly?

[jakub-c]

I've been struggling to mount something else than default \Users directory on Mac OS X, and here's what worked for me:

• Go to Virtual Box app
• Choose the docker VM in my case it's called 'default'
• Go to Settings -> Shared Folders
• In the Folder List double click an item under 'Machine Folders' - that's the folder docker is going to use. By default it's going to be aliased as 'Users' unless you change it

So in your docker run you'll be able to do -v /Users or whatever name you've set up.

It's important to set it up under 'Machine Folders' not under 'Transient Folders' in order to make it work.

[posita]

@jakub-c, thanks for the VBox-specific work-around, but I don't think that's the point of this issue. As far as I understand, this is a feature request to abstract (or at least expose) each underlying drivers' shared folder mounting options through the docker-machine command.

[agentfitz]

Just getting started with Docker (from the Vagrant world) and this has been my biggest pain point. An easy and intuitive way to sync a host directory (from OSX) all the way down to the web root in the container would be HUGE.

[fesor]

@agentfitz for OSX there is dinghy project

[agentfitz]

@fesor Thanks! Hey, last night I realized I had totally missed something. Since /Users is automatically mapped from OSX to the docker-machine (if I'm not mistaken), then setting up your synced volume from your host OSX all the way down into the docker container is as simple as targeting a specific directory from within /Users for your mapping, so:

docker run -v /Users/my/folder:/var/www me/my-docker-image

I actually got this working last night after a hidden tip on Stackoverflow. Is this clearly laid out in the Docker docs somewhere? Also, considering how easy this is (once you know how it works), why do all these other syncing projects exist?

[mrmachine]

@agentfitz because VirtualBox is terribly slow at sharing folders from OSX into the VM. Even VMware, which is much faster, is still supposedly not as fast as NFS (which dinghy uses), which is again not as fast as rsync (which docker-osx-dev uses). Rsync has the drawback of actually copying files into the container, though, which often may not be practical. Plus, there are permissions issues. Host volumes mounted into the container will have the same UID/GID as the host. If the UID/GID of the process running inside the container doesn't match, you could have issues.

[krasi-georgiev]

https://github.com/adlogix/docker-machine-nfs

is by far the most simple solution and my favourite.
It is just a script which sets the nfs share even after the machine is creates.

[agentfitz]

@mrmachine ah, so are you saying the -v flag to the docker run command uses VirtualBox's built in syncing under the hood? What is the "syncing technology" it uses? I do remember reading about performance concerns but when I set it up last night source code changes seemed to be reflected in the docker container instantly. Is it an issue with bigger files (like images or something?) as opposed to source code? Thank you for your response!

I also hadn't considered the permissions issues, so thanks for pointing those out. @vipconsult thanks for sharing, I'll check it out.

But seriously though, where are the official dockers docs on all this?

[nathanleclaire]

ah, so are you saying the -v flag to the docker run command uses VirtualBox's built in syncing under the hood?

-v flag for docker run specifies that the first part of the arg (before the colon) should be bind mounted into the container at the second part of the arg. It _happens_ to be the case that we mount /Users from the host (OSX) into /Users in the VM (Linux / boot2docker) at boot of the VM.

What is the "syncing technology" it uses? I do remember reading about performance concerns but when I set it up last night source code changes seemed to be reflected in the docker container instantly. Is it an issue with bigger files (like images or something?) as opposed to source code?

VirtualBox shared folders -- they work fine for simple use cases, but with large files and/or lots of reads and writes their performance is not ideal.

[Compufreak345]

@agentfitz For me the performance issues are huge as I use SQLite-databases. I am also using the script mentioned by vipconsult since some months, it works really well.

[joergpatz]

+1 for adding more driver options for sharing folders. would be great to see this added to docker-machine.

[erdnaxeli]

+1

[RaeesBhatti]

bump...

[RaeesBhatti]

A method recommended by @jas99 worked for me.

[kevinpapst]

+1

[MrLeit]

+1

[walterdolce]

+1 to having the ability to configure shared folders outside the default /Users folder. It would be great to see this currently missing functionality be prioritised, implemented and released so people don't have to resort to this: https://github.com/docker/machine/issues/1826#issuecomment-248436823.

[nathanleclaire]

+1 to having the ability to configure shared folders outside the default /Users folder. It would be great to see this currently missing functionality be prioritised, implemented and released so people don't have to resort to this: #1826 (comment).

This _could_ mean a lot of different things, but FWIW, it'll be possible in 0.9.0 thanks to @u-foka !

[walterdolce]

Nice! Thanks @nathanleclaire and @u-foka! 🚀

Breaking down what I said above:

having the ability to configure shared folders outside the default /Users folder

To:

outside the default /Users folder

That means having the ability to allow docker to see the following (note there are _multiple_ folders, not just an "alternative" like the initial comment in #1826 says):

/Users
/Volumes
/whatever

Basically what Docker for Mac does:

[nathanleclaire]

going to close this issue in favor of more specific ones. "should be configurable" is overly broad. thanks all!