Docker-sync: git throws permission denied when docker-sync is running (Windows)

[mrmkrs]

Error

It seems that docker-sync / unison doesn't work well with git. I was experiencing issues with PHPStorm not being able to pull anymore when I had some commit ready to be pushed. I managed to reproduce it using a simple setup described below. The error is that git somehow wants to get access to a file, but cant due to permission errors.

Docker Driver

Windows 10, Docker for Windows WSL

Sync strategy

Unison

your docker-sync.yml

version: "2"
options:
  verbose: true
syncs:
  wsl-demo-sync: # tip: add -sync and you keep consistent names als a convention
    sync_args:
      - "-perms=0" #required for two way sync ie generators, etc
      - "-fastcheck=true" #required for two way sync ie generators, etc
      - "-ignore='Path .git'"           # ignore the main .git repo
      - "-ignore='BelowPath .git'"      # also ignore .git repos in subfolders such as in composer vendor dirs
    sync_strategy: 'unison'
    sync_host_ip: '127.0.0.1' #host ip isn't properly inferred
    sync_host_port: 2375
    src: './docker-sync'
    sync_userid: '33'

OS

Windows 10

project setup

To be done on Windows:

1. Place docker-compose.yml and docker-sync.yml in an empty folder (C:\TEMP\unison in my case)
2. git.exe clone --progress -v "[email protected]:EugenMayer/docker-sync.git" "C:\TEMP\unisondocker-sync"

To be done in WSL:

1. cd /c/TEMP/unison
2. docker-sync start
3. docker-composer up -d

To reproduce

In windows:

1. git.exe reset --hard ab2b9c4697f15dad86d53a79f424c4589789de14
2. change a file, for example README.md
3. commit the change
4. git.exe pull --progress -v --no-rebase "origin"

Response I got:

From github.com:EugenMayer/docker-sync
= [up to date]      master     -> origin/master
= [up to date]      gh-pages   -> origin/gh-pages
error: cannot stat 'spec/fixtures/app_skeleton/src': Permission denied
error: cannot stat 'spec/fixtures/native_osx': Permission denied
error: cannot stat 'spec/integration': Permission denied
error: cannot stat 'spec/integration': Permission denied
error: cannot stat 'spec/lib/docker-sync/dependencies/package_managers': Permission denied
error: cannot stat 'spec/lib/docker-sync/dependencies/package_managers': Permission denied
error: cannot stat 'spec/lib/docker-sync/dependencies/package_managers': Permission denied
error: cannot stat 'spec/lib/docker-sync/dependencies/package_managers': Permission denied
error: cannot stat 'spec/shared_examples': Permission denied
error: cannot stat 'spec/shared_examples': Permission denied
error: cannot stat 'spec/shared_examples': Permission denied


git did not exit cleanly (exit code 128) (2015 ms @ 17-7-2017 15:19:28)

If I do a docker-sync stop in WSL, and try again it does work.

[EugenMayer]

@rwilliams could you be able to comment on this?

[rwilliams]

@mrmkrs @EugenMayer I'm a little busy with a few things atm but later this week I can start looking into this in detail.

[mrmkrs]

Thanks.
Here's the docker-compose.yml I used for the test:

version: '2'

services:

    code:
        image: alpine:latest    
        volumes:
            - wsl-demo-sync:/opt/project:nocopy
volumes:
    wsl-demo-sync:
      external: true

[mrmkrs]

Just in case it meight be helpfull, here's the output when starting docker-sync

marcel@gulpener:/c/TEMP/unison$ docker-sync start
/var/lib/gems/2.3.0/gems/docker-sync-0.4.6/lib/docker-sync/update_check.rb:42: warning: Insecure world writable dir /mnt/c in PATH, mode 040777
          ok  Starting unison for sync wsl-demo-sync
          ok  wsl-demo-sync container not running
          ok  starting wsl-demo-sync container
     command  docker start wsl-demo-sync && docker exec wsl-demo-sync supervisorctl restart unison
          ok  starting initial sync of wsl-demo-sync
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' wsl-demo-sync
     command  unison -testserver /app_sync "socket://127.0.0.1:32775"
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' wsl-demo-sync
     command  unison '/c/TEMP/unison/docker-sync' -auto -batch -prefer '/c/TEMP/unison/docker-sync' -copyonconflict -perms=0 -fastcheck=true -ignore='Name *.log' -ignore='Path .idea' -ignore='Path .git' -ignore='BelowPath .git' socket://127.0.0.1:32775
          ok  Synced /c/TEMP/unison/docker-sync
      output  - bin
     success  Unison server started
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' wsl-demo-sync
     command  unison '/c/TEMP/unison/docker-sync' -auto -batch -prefer '/c/TEMP/unison/docker-sync' -copyonconflict -perms=0 -fastcheck=true -ignore='Name *.log' -ignore='Path .idea' -ignore='Path .git' -ignore='BelowPath .git' socket://127.0.0.1:32775
          ok  Synced /c/TEMP/unison/docker-sync
      output  \ bin
     success  Starting Docker-Sync in the background
marcel@gulpener:/c/TEMP/unison$

So just to be clear, the only commands executed in WSL are docker-sync start and docker-compose up. Every other action (including git) is done in Windows.

[rwilliams]

Can you try running git in WSL to see if you get the same issue? I think it might just be a WSL interop quirk.

[mrmkrs]

Yup that works perfectly without errors. Hope you're not going to advice a Windows user to do git stuff on the command line from now on :)

[rwilliams]

@mrmkrs This stack overflow article might be relevant. Unison is probably interfering with git. Maybe we can use the Windows version of unison instead of the Linux version?

[EugenMayer]

@rwilliams might be troublesome due to probably missing inotify events or "not as good" events.

I would not really like to add a other unison to the complexity matrix, if not needed

[EugenMayer]

Reading that thread and imagine what they actually do write there - its completely nonsense to use Windows for such kind of development. Close your IDE to be able to do git operations?

That would mean that any open file handle, what ever kind, could cause this... that sounds like a gate to hell here

[mrmkrs]

Well is does sometimes happen that process locks a file. But I never ever had to close my IDE to perform a GIT operation.
Nevertheless I uninstalled TortoiseGIT to test if anything improves but no. Upgraded git version, no luck there. The strange thing is that if I do a git pull (in windows) the very first time after docker-sync start (in wsl), no errors! But, to a git reset, followed by a git pull and the error is back. Looks like some strange timing thing although the same set of files are 'locked' every time. And docker-sync logs never ever mentions these files.

The only viable workaround for now is to 'docker-sync stop', git pull, docker-sync start. Due to the enormous speed improvement docker-sync provides, I'm willing to accept that. But it's not ideal.

Just for fun I did install unison on windows (not wsl), installed latest ruby and did a docker-sync. But who would expect that to work :) I don't know ruby and havent looked at your sourcecode but based on the unison website I expect unison to watch the filesystem? So what does docker-sync do? Generate and execute all unison / docker commands?

Any ideas on what I could do to give this another go?

[rwilliams]

@mrmkrs You can add arbitrary windows directories to your $PATH in WSL. Try adding the windows binaries for unison and unison-fsmonitor to the WSL $PATH and running docker-sync from WSL. You'll also probably have to remove or rename the WSL unison from your $PATH.

[mrmkrs]

Tried it, but it now complains about pathnames. Funny thing is that if I copy the unison command into Git Bash it does sync...

marcel@gulpener:/c/TEMP/unison$ docker-sync start
/var/lib/gems/2.3.0/gems/docker-sync-0.4.6/lib/docker-sync/update_check.rb:42: warning: Insecure world writable dir /mnt/c in PATH, mode 040777
          ok  Starting unison for sync wsl-demo-sync
          ok  wsl-demo-sync container not running
          ok  creating wsl-demo-sync container
     command  docker run -p '127.0.0.1::5000' -v wsl-demo-sync:/app_sync -e VOLUME=/app_sync -e TZ=${TZ-`readlink /etc/localtime | sed -e 's,/usr/share/zoneinfo/,,'`} -e UNISON_EXCLUDES="" -e OWNER_UID="33"  --name wsl-demo-sync -d eugenmayer/unison
          ok  starting initial sync of wsl-demo-sync
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' wsl-demo-sync
     command  unison -testserver /app_sync "socket://127.0.0.1:32786"
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' wsl-demo-sync
     command  unison '/c/TEMP/unison/docker-sync' -auto -batch -prefer '/c/TEMP/unison/docker-sync' -copyonconflict -perms=0 -fastcheck=true -ignore='Name *.log' -ignore='Path .idea' -ignore='Path .git' -ignore='BelowPath .git' socket://127.0.0.1:32786
       error  Error starting sync, exit code 0
     message
     message  Contacting server...
Fatal error: Cannot find canonical name of /c/TEMP/unison/docker-sync: unable to cd either to it

(/c/TEMP/unison/docker-sync: No such file or directory)
or to its parent /c/TEMP/unison
(/c/TEMP/unison: No such file or directory)
     success  Unison server started
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' wsl-demo-sync
     command  unison '/c/TEMP/unison/docker-sync' -auto -batch -prefer '/c/TEMP/unison/docker-sync' -copyonconflict -perms=0 -fastcheck=true -ignore='Name *.log' -ignore='Path .idea' -ignore='Path .git' -ignore='BelowPath .git' socket://127.0.0.1:32786
       error  Error starting sync, exit code 0
     message
     message  Contacting server...
Fatal error: Cannot find canonical name of /c/TEMP/unison/docker-sync: unable to cd either to it

(/c/TEMP/unison/docker-sync: No such file or directory)
or to its parent /c/TEMP/unison
(/c/TEMP/unison: No such file or directory)
     success  Starting Docker-Sync in the background
Creating network "unison_default" with the default driver
Creating unison_code_1
marcel@gulpener:/c/TEMP/unison$

[rwilliams]

I'm not sure where to go with this one. I think it's a weird WSL interop quirk. Targeting git-bash as a first step towards native Windows support seems like a good idea.

[mrmkrs]

So I've been using docker-sync over the past week, tried to tweak some stuff but I'm going to call it quits. The syncing stuff works really well and performance is really good, but combining it with my daily GIT operations just kills productivity.

If you could find a strategy where you can factor out WSL, my guess is that things could improve. I'd be happy to test things if needed. There is also a few ms (100-500ms) lag between the file change and sync action, where you sometimes wonder if changes made it to the container in time before the browser refresh.

The alternative I've setup now is adding an additional SSH container and configured phpstorm to 'deploy' to that container when files change.

[rwilliams]

@mrmkrs I hear you. My plan is to target git-bash when I get time. It should be doable and WSL just adds way too much complexity imo.

[EugenMayer]

@rwilliams @mrmkrs should we close this issue for now, until you guys dedicate time for it, or are you planning to go for this any time soon?

[mrmkrs]

I'll be happy to test this again when changes are available (for as long I'm not switching to linux) :)

[EugenMayer]

I will close this issue for now, since i think its drifted away. Of course, open it anytime you want to continue working on this!

[tim-arnold]

@EugenMayer

We have 10 developers running Docker for Windows and Docker Sync, and this issue is the most vexing we face. It doesn't happen all the time, but enough to be a real problem. Honestly, it's the main reason we are looking at Linux machines for our next purchasing cycle, but that's still more than a year away.

When the issue rears its head, we cannot pull from remote repos or change branches without first stopping docker-sync. The result is frustration and wasted time waiting for things to come back up. Any chance this one will drift back under your spotlight?

[EugenMayer]

@tim-arnold well, i am not using Windows for development at all and i also opted out that i will support windows myself here - @rwilliams opted in to do so.. i guess time passed.

Beside that, i cannot make up what the case could be. Sure Windows has its own locking mechanism on files with NTFS, or its the tortoisn / alikes GUI that does that, i do not know. To many factors/variables for me. That said, since you have 10 devs using this, you might take a glimps at this yourself?

https://github.com/EugenMayer/docker-sync/wiki/6.-Development
https://github.com/EugenMayer/docker-sync/wiki/8.-Strategies#unison-osxwindowslinux
https://github.com/EugenMayer/docker-sync/wiki/docker-sync-on-Windows

You will mostly need windows tools, trying to understand how locking works for the tools you are using. The point here is, that you understand how docker-sync works under windows.

Means, its not the same as the native_osx strategy. What basically is done is:

1. /usr/local/bin/unison-fsmonitor is used to watch for changes while ON your WSL
2. then it triggers /usr/local/bin/unison (when changes are found) and then
3. that unison then syncs the folders into a docker-container ( sync container ) to a unison instance running there (its counterpart)
4. ....

You do not need to go further then 4 with your debugging, since that cannot be your issue. So what we are asking ourself is:

a) How dows WSL get the FS from your windows in terms that you have your windows files in there ( never used WSL )
b) does /usr/local/bin/unison-fsmonitor somehow "lock" the files in a way, that nobody is allowed to change them ( git branch ). It somehow makes 0 sense, this is a FS even watcher, why should that be the case (ever)

Assumption:

• you change branch, the first file changes flying int, unison-fsmonitor registrers an triggers unison .. this starts syncing, means it will read the files. This might lock them ( for read ) and blocks your git operation to further proceed changes the folder whatever.

So, thats as far as i can get you, no comeon, help me out :)

[mrmkrs]

For what it's worth, as it doesn't aid in this issue: I switched from Windows to Linux (kubuntu 17) since december, and I'm still trying to figure out why I hadn't done that a year ago. You just instantly "feel" docker is meant to be run on Linux. Or better: not meant to be run on Windows. And this comes from the second to last guy at my company that switched from Windows to Linux. IMHO this is a great project, but if you want true productivity with development in Docker, use OSX or Linux.

[EugenMayer]

closing issue, since there is no progress made / work done. reopen anytime you start again

[bfredtim]

I meant to follow up after @EugenMayer so kindly responded with more suggestions than he probably even needed to. Thanks! We have started running Docker in Ubuntu 16 VMs and life is a complete breeze now. Like @mrmkrs , I wish we'd just done this at the very start. Docker for Windows is just not ready for real work as far as we can tell.

[EugenMayer]

I can imagine that those things are even worse as onder OSX, but well, i guess whatever works for you