Docker-sync: Not syncing both directions suddenly

[nurtext]

Error/Feature Requestion/Docs

All docker container are up, but not syncing between /host_sync and /app_sync.
/app_sync to /host_sync is working.

Docker Driver

d4m

Sync strategy

native_osx

docker-compose.yml

version: '2'

services:
    api:
        container_name: api
        image: api
        build: .
        ports:
            - 8888:80
        volumes:
            - ./app:/srv/api/app
            - ./bin:/srv/api/bin
            - ./src:/srv/api/src
            - ./tests:/srv/api/tests
            - ./web:/srv/api/web
            - ./vendor:/srv/api/vendor
        environment:
            DATABASE_HOST: "127.0.0.1"

docker-compose-dev.yml

version: '2'

services:
    api:
        volumes:
            - api-app-sync:/srv/api/app:nocopy
            - api-bin-sync:/srv/api/bin:nocopy
            - api-src-sync:/srv/api/src:nocopy
            - api-tests-sync:/srv/api/tests:nocopy
            - api-web-sync:/srv/api/web:nocopy
            - api-vendor-sync:/srv/api/vendor:nocopy

volumes:
    api-app-sync:
        external: true

    api-bin-sync:
        external: true

    api-src-sync:
        external: true

    api-tests-sync:
        external: true

    api-web-sync:
        external: true

    api-vendor-sync:
        external: true

docker-sync.yml

version: '2'

options:
    verbose: true

syncs:
    api-app-sync:
        src: './app/'

    api-bin-sync:
        src: './bin/'

    api-src-sync:
        src: './src/'

    api-tests-sync:
        src: './tests/'

    api-web-sync:
        src: './web/'

    api-vendor-sync:
        src: './vendor/'

docker ps output

CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                            NAMES
72ee4eb5a02b        api                              "docker-php-entryp..."   2 minutes ago       Up 2 minutes        9000/tcp, 0.0.0.0:8888->80/tcp   api
7ec87ccaa84f        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   15 minutes ago      Up 2 minutes        500/tcp                          api-vendor-sync
de22a140b897        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-web-sync
b1d749f9c992        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-tests-sync
62d9a55979a3        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-src-sync
0af351c80ed8        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-bin-sync
45d5f50d0659        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-app-sync

docker-sync list output

      ok  Found configuration at /Users/cedric/api/docker-sync.yml

api-app-sync On address :
api-bin-sync On address :
api-src-sync On address :
api-tests-sync On address :
api-web-sync On address :
api-vendor-sync On address :

OS

macOS 10.11.6

[michaelbaudino]

Hi @nurtext,

First, thanks for the detailed description 👍

Could you please add the output of docker logs api-web-sync ? (or whatever *-sync container you're trying to sync files with)

[codycraven]

I've ran into this intermittently since going to native_osx. There is nothing visible in docker logs that indicates a problem, however I suspect it stems from timestamps being out of sync between the host and the container.

In my local configs I'm now applying volumes to my containers that use docker-sync volumes like so:

    api:
        volumes:
            - /etc/localtime:/etc/localtime:ro # Fix TZ issues w/ docker-sync?
            - api-app-sync:/srv/api/app:nocopy
            - api-bin-sync:/srv/api/bin:nocopy
            - api-src-sync:/srv/api/src:nocopy
            - api-tests-sync:/srv/api/tests:nocopy
            - api-web-sync:/srv/api/web:nocopy
            - api-vendor-sync:/srv/api/vendor:nocopy

I don't know how to reproduce the issue so I'll need to see overtime if this "fix" prevents the state from occurring.

Update: This doesn't work.

[EugenMayer]

@codycraven so the issue was your time? (does it work now?)

since timestamp compares are used to detect if a file has been changed, that could be cause

[codycraven]

@EugenMayer thus far I have not encountered sync failing since implementing this earlier today.

My PR for my team was merged about an hour ago, so I should know by mid-week next week if this solved it (most of our team is traveling tomorrow and Monday is a holiday so I won't have a good sample until then).

[EugenMayer]

Very interesting, lets see how this turns out @codycraven - keep us posted. We do handle timezones: https://github.com/EugenMayer/docker-sync/blob/master/lib/docker-sync/sync_strategy/native_osx.rb#L86 in the sync container.

So if your app container does not do that and your TZ differs, that is a good source for issues. This leaves us with the question, how we could solve this for the docker-sync user - seems like we can only do that by documentation?

[michaelbaudino]

Ho, yes, this is very confusing 😕

→ date && docker exec frontend-sync date && docker-compose run frontend date
Fri May 26 10:52:41 CEST 2017
Thu May 25 20:32:38 CEST 2017
Thu May 25 18:33:07 UTC 2017

On MacOS Sierra 10.12.4.

[EugenMayer]

Thats a very common docker issue its basically there always, just not always visible. but since we compare timestamps on 2 different containers, that happens

[azamat163]

Hi guys!
I also encountered such a problem, but solved it with the help of the Mac OS update to 10.12.5. It helped me.

[EugenMayer]

That is pretty odd, how does this get fixed by an Mac OS update .. were did you had this hint from? Which OS did you had before, El Capitan?

[azamat163]

Yes,10.11.4

[EugenMayer]

@codycraven are you / your collegues on El Capitan?

[codycraven]

@EugenMayer I'm on Sierra. Looking at the native_osx.rb Docker run command is what gave me the idea to sync the time zone. Going a step further (if this doesn't resolve it) a colleague had the idea to sync the containers (VM) and host with NTP.

[nurtext]

Same timezone on host and all docker containers, problem still remains:

> date && docker exec api date && docker exec api-app-sync date
Mon May 29 11:39:34 CEST 2017
Mon May 29 11:39:34 CEST 2017
Mon May 29 11:39:34 CEST 2017

Logfile:

> docker logs api-app-sync
2017-05-29 11:29:05,061 CRIT Supervisor running as root (no user in config file)
2017-05-29 11:29:05,062 WARN Included extra file "/etc/supervisor.conf.d/supervisor.daemon.conf" during parsing
2017-05-29 11:29:05,097 INFO RPC interface 'supervisor' initialized
2017-05-29 11:29:05,097 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2017-05-29 11:29:05,097 INFO supervisord started with pid 1
2017-05-29 11:29:06,100 INFO spawned: 'unison' with pid 27
2017-05-29 11:29:07,271 INFO success: unison entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Edit:
Maybe this is another puzzle piece, but I doubt so: I recently moved from ubuntu:trusty-20170214 to php:7.1-fpm-alpine. Previous setup was working fine with native_osx sync

[michaelbaudino]

I discovered something that may help: my team leader (who's running the exact same Docker Compose and Docker Sync config than I do) is impacted by this problem and using @codycraven's workaround to share /etc/localtime fixed the issue for him (thanks for that, @codycraven, btw 🎩 ).

The only difference I can think of is that my containers and host seem totally desynchronized while his containers seem to have more consistent times:

→ date && docker exec frontend-sync date && docker-compose run frontend date
Mon 29 May 2017 16:49:06 BST
Mon May 29 16:49:06 BST 2017
Mon May 29 15:49:18 UTC 2017

[EugenMayer]

@michaelbaudino are you using any "--prefer newer" syntax?

[michaelbaudino]

Nope: we use the default. Here is our exact config: https://gist.github.com/michaelbaudino/1c83c253794e5b6bd0cc8fc47c1b4951

[EugenMayer]

@michaelbaudino what about OS differences, are you running Sierra and he is running El Capitan?

Still trying to understand the pattern here

[michaelbaudino]

Nope, we are both on Sierra 10.12.4.

Actually, we were, because I upgraded to 10.12.5 about 5 minutes ago 😄

[codycraven]

@michaelbaudino when you added the TZ fix had you destroyed your old containers/volumes before bringing them back up or just stopped them?

[codycraven]

I just ran into this again on my machine. The time is synchronized perfectly so I continued digging.

I checked the file I modified in my host within the docker-sync container and was astounded to find that the file within /host_sync/ was the old version that does not match what is on my host.

Based on this, it appears that the volume mount into the docker-sync container is somehow breaking.

Update
This appears to be a false failure. I created a copy of my project in a new directory and ran it while debugging something. I failed to properly clean up afterwards as my sync container's bind mount was pointed at my copied directory instead of the one I was working on.

[masev]

I have exactly the same issue. I updated to the last version of Sierra and tried the hack to share localtime with no luck ...

Tell me if I can help you by giving more info.

[EugenMayer]

Ok so what we have right now is that:

a) the timezone does matter - but not for everyone.

b) Sierra or el Capitan are not significant also - both could have this issue

c) there is no pattern in the project configuration or anything too special

---

Bottom line is, that we need to draw a tighter circle around the issue, it's far too broad right now. Suggestions:

I) we write a wiki guide on how to debug issues or gather information - this is overdue anyway. So compare host/app any, check /tmp/unison
. How to enter a snyc container, how to manually restart the inner unison project

II) we gather information from all people having this issue and all not having this one and see, were we have important differences - I would suggest a Google spreadsheet for this probably

Anybody willing to help bootstrapping this?

[nurtext]

@EugenMayer Since I'm depending upon a working docker-sync for the next release of my project, I'm willing to help. Tell me what's up next and I'll do my best.

[codycraven]

I'm in, I can start work on a debug guide tomorrow (or contribute to it if someone else starts one).

[nurtext]

Some more debugging for you:

Cleaned up everything: intermediate containers, docker-sync-stack, containers, etc. No change at all.

Changed a file on host: no result.

1. docker exec -it api-app-sync /bin/bash
2. ps aux
3. kill -9 $PID_OF_UNISON

Changed file got recognized:

Contacting server...
Looking for changes
Reconciling changes
changed  ---->            Resources/views/root.html.twig
Propagating updates
UNISON 2.48.4 started propagating changes at 09:52:26.44 on 31 May 2017
[BGN] Updating file Resources/views/root.html.twig from /host_sync to /app_sync
[END] Updating file Resources/views/root.html.twig
UNISON 2.48.4 finished propagating changes at 09:52:26.45 on 31 May 2017
Saving synchronizer state
Synchronization complete at 09:52:26  (1 item transferred, 0 skipped, 0 failed)
Looking for changes

Tried to change a file afterwards: Same behavior and no more changes got recognized.

Result of ps aux:

PID   USER     TIME   COMMAND
    1 root       0:00 {supervisord} /usr/bin/python /usr/bin/supervisord
   16 root       0:00 tail -F /tmp/unison.log
   24 root       0:00 /bin/bash
   68 root       0:00 tail -f unison-stdout---supervisor-T9b2x9.log
   69 root       0:00 /bin/bash
   75 root       0:00 unison -prefer /host_sync -numericids -repeat watch -auto -batch /host_sync /app_sync -logfile /tmp/unison.log
   76 root       0:00 /usr/local/bin/unison-fsmonitor
   79 root       0:00 ps aux

[EugenMayer]

@nurtext @codycraven great. First - lets put in the guide in place, since that will be the first thing to ensure we are all doing reproductionable steps and are all on the same way - the only way to really find the actual issue pattern. Could you help with https://github.com/EugenMayer/docker-sync/issues/415

@codycraven if you start the guide, i join you adding what i know about the containers, images, supervisord, debugging and things like that - please all chats about this further in https://github.com/EugenMayer/docker-sync/issues/413 - i created a first outline of what we should do and why, feel free to actually be bold in any sense - i just emptied my mind there

@michaelbaudino Mr. Spec, anything we could do with spec to formalize testing? I guess we would need to start with the bash thingy and it will take longer then we want to get this thing on the line, right? Created https://github.com/EugenMayer/docker-sync/issues/414 for this - lets see

Please try to spread the comments for the topics across the outer issues - the only thing we should do in this tickets is giving analysis results and give people a spot to "i am also effected" - and also advertise debugging steps / new things we want to know to get it fixed.

Lets get our hands dirty on this - i see that is a serious issue here and we need to get this done ASAP, but i will need assistance to get it into a timeframe it suits you.

[nurtext]

Some more debugging according according to the guide in the wiki…

Host Sync working:

> diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/host_sync/$DEBUG_DOCKER_FILE")
> 

App Sync not working:

> diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/app_sync/$DEBUG_DOCKER_FILE")
Files Resources/views/root.html.twig and /dev/fd/11 differ

Log is empty:

> docker exec "$DEBUG_DOCKER_SYNC" tail -n70 /tmp/unison.log
> 

Missing directory:

> docker container inspect --format '{{(index .Mounts 1).Source}}' "$DEBUG_DOCKER_SYNC"
/var/lib/docker/volumes/api-app-sync/_data
> docker exec "$DEBUG_DOCKER_SYNC" ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directory
> ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directory

Will now try to reset d4m…

[EugenMayer]

No log means no unison is actually starting @nurtext, please see the supervisor logs for the reasons that this does not actually work or try to restart unison:

supervisorctl restart unison

[nurtext]

@EugenMayer Resetting d4m helped. Sync is working again. I guess this won't help finding the real cause of the issue, but here's a clue:

Before resetting d4m:

> docker container inspect --format '{{(index .Mounts 1).Source}}' "$DEBUG_DOCKER_SYNC"
/var/lib/docker/volumes/api-app-sync/_data

After resetting d4m:

> docker container inspect --format '{{(index .Mounts 1).Source}}' "$DEBUG_DOCKER_SYNC"
/Users/cedric/Git/Project/api/app

This is the correct path indeed and I don't know why Docker was thinking that /var/lib/docker/... was the correct path.

[EugenMayer]

the latter path is the local path, i guess the one you posted first is from the xhyve VM - but yes, that one is odd and still interesting. @nurtext you actually have been written host_sync is working, right - that makes this even less likely, since that wrong host paths would result in host_sync being not synced ..

The other way is, that the files are synced but due to a symlink or wrong path above fs-events have not been triggered. Still pretty dark

[nurtext]

@EugenMayer
Very strange indeed since app_sync -> host_sync was working a few days ago as I started this topic. Never tested again since then…

However I guess fs-events weren't triggered, thus no sync has been made. This could be an explanation for the host_sync -> app_sync working after killing / restarting unison: https://github.com/EugenMayer/docker-sync/issues/410#issuecomment-305114984

[jenswegar]

Experiencing this same issue it seems.

System: macOS Sierra 10.12.5
docker sync 0.4.6
Docker for Mac 17.03.1-ce-mac12 (17661) [newest version at time of writing]

host sync seems to work (no output from command below):

> diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/host_sync/$DEBUG_DOCKER_FILE")
> 

app sync seems to fail:

> diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/app_sync/$DEBUG_DOCKER_FILE")
> Files web/themes/custom/xyz_2017/css/style.css and /dev/fd/63 differ

log is empty,even after file change on host:

> docker exec "$DEBUG_DOCKER_SYNC" tail -n70 /tmp/unison.log
> 

running docker-sync clean, rebuilding my local images for the affected project (docker-compose build) and restarting Docker for Mac seems to have corrected the issue for now.

[codycraven]

Interesting tidbit - I had two co-workers run into this problem on two different days. The ultimate solution ended up being a down, docker-sync clean, shut down, then re-start and up.

I'm guessing we're running into some sort of limit where the Unison process can no longer track changes. I'm trying to learn more about Unison to figure out what it could be.

[jenswegar]

This might be of interest: Today the sync started to act up again (stopped to sync from host to app), but this time all I did was restart the Docker for Mac daemon. However, when I just tried to use the "restart" button in the Docker for Mac toolbar menu, the restart just hanged itself (never completed). I had to use Docker for Mac menu -> Preferences -> Daemon -> Advanced and there the "reset your daemon settings" option for the restart to go through.

So maybe the issue with the sync is that some process hangs itself in the background. Does native_osx sync use rsync somehow? Because I've heard from colleagues who work with backups in the Qnap world that rsync has a tendency to go ape shit if synching very large amounts of files (what large amounts means remained unclear still)

[donnykurnia]

@jenswegar native_osx using unison. Docker for Mac indeed can hang while restarting, but it's not only because of docker-sync. If your Mac runs up the resources, Docker for Mac will be swapped out from memory and this will cause the daemon getting slower. See the resources usage using Activity Monitor.

[eggie5]

this was just happening to me post native-mac sync strategy update: sync was only working 1-way. The fix for me like someone noted above was: docker-sync clean

[hyviquel]

I'm noticing similar problems as well on my macbook since I'm using native-mac sync strategy... Let me know if you want me to test anything.

[Cethy]

I, too, seems to have this problem regularly on osX (macbook), seemingly related to a memory usage issue.

I managed to unblock it with a simple "manual" docker-sync sync.

[michaelbaudino]

Just to add some content to the current thoughts, this bug bit me again yesterday while setting up our development environment on a new team member MacOS Sierra 10.12.5 (with Docker 17.06.0-ce): synchronization (using native_osx) was simply not working in any way (neither from container to host, nor from host to container).

Running docker-sync stop; docker-sync clean; docker-sync start did not fix it (we tried twice). Restarting the Docker engine did fix it, though 🤷‍♂️

Anyway, the real interesting this I noticed was that changes on host (made using Visual Studio Code and nano with the same results) were successfully propagated to /host_sync but not to /app_sync. I tried to monitor the FS events (using inotify -r -m /host_sync from inside the sync container) and I was surprised to see that changes on host were correctly applied in /host_sync but without any FS event being triggered 😲 (which explains why Unison did not see anything and thus did not propagate the changes to /app_sync).

I know that FS events in osxfs is rather incomplete, but that's not the problem here: we were using a (supposed to be) supported FS event (modification event).

I also saw docker/for-mac#681 that might be related and is in progress to be fixed by moby/moby#33048 (and its implementation in moby/moby#33375) which I find very promising 😊.

Final note: I cannot ask my team members to stay in a state where "the bug can be replicated but they cannot work" obviously, so I was happy when it magically auto-solved by restarting the Docker engine and thus I still do not know how to reproduce consistently or what is the cause factor 😞

[EugenMayer]

@michaelbaudino thank you for finally confirming what i was suggesting all the times.

Since all people reported a running host sync and all people reported that restarting the stack does not help AND restarting the inner unison does not help, it was the only option left, failing FS-Events on the linux container, not properly propagated when syncing with OSXFS.

great you found an actual issue, i will try to join the discussion there.

Since a mac restart is actually not needed, but a docker-engine restart seems to fix it, it is a plain docker for mac issues, no strings attached.

That at least for now is an outcome - but also one we cannot really influence/autodetect and quick fix. We could put some effort into actually trying to do test-changes in a special host mount, like a health check container, and if we do not see any events, we inform the user, but that is a huge amount of effort and will just not fix it, but rather make it transparent.

Not sure how we could proceed here - its clear that this bug does only happen for native_osx, since the unison strategy does not rely on OSXFS and its FS events, but rather on OSX fsevents, with the huge drawback in performance during watching the codebase.

[michaelbaudino]

Well, fixing Docker For Mac is the very essence of Docker Sync, isn't it? 😜

[huersch]

sorry for opening another issue. I didn't recognise this one.

I can validate that restarting docker for mac at least temporally fix this issue and also I do have different time zones on the docker-compose container compared to the host and sync container.

Sadly we can reproduce the issue on several devices. I have the feeling that it always works the first time you start a sync.

If I can do anything of help, please let me know. Since several colleagues have this problem and this is pretty crucial for us, we will investigate further and will let you know if we find something new.

[michaelbaudino]

@huersch Thanks for your report.

Something that would indeed help would be to find the factor that lets us reproduce the bug on demand. So far, we've identified some symptoms, but not the reproduction factor 😕

[EugenMayer]

That won't be easy, since it might be hidden in the filesystem implementation of OSXFS. We might play around with ulimit values, since I expect this bug to correlate with the amount of file changes per time span. One of the common preconditions to force the bug is, changing branches or generating assist with gulp and alikes. Also i smell an indication that the performance of your Mac does influence the probability .. it seems higher performance lets you avoid it.

Combining that means, that we might see a queue which fills up with change events, e.g. in OSXFS, when this one is not worked through fast enough, we hit limits and processesing of the queue stalls completely, probably to e.g. to many open file handles. Rrstarting containers and docker sync does.not help as expected, since the queue is part of the osxfs layer between xhyve and osx... Rrstarting d4m helps since this hard flushes the queue.

This issue might in fact be related to the xhyve VM and limits in there. We need to find a way to analyze the VM logs and fsstats when the issue happens

[huersch]

Do you think changing sync strategy to rsync would fix it temporally? We just need to sync files from host to app.

[EugenMayer]

It will fix one thing and introduce a LOT more CPU load on the host due to the fswatch watchers - its a tradeof

[mf-lit]

One of the common preconditions to force the bug is, changing branches or generating assist with gulp and alikes.

Yes, these are two circumstances where we are definitely seeing the problem. I can reliably reproduce the issue by leaving a loop running that keeps checking out different branches, meanwhile I'll start changing some test files.

[btaylor-ihr]

Eugen asked me to pass on that restarting my Docker appears to have fixed the issue. I jumped on https://github.com/EugenMayer/docker-sync/issues/452 and said I have the same issue

My docker-sync was dying at least once every day. I was traditionally destroying the volume and syncing again with a fresh volume, but that took anywhere up to 40 minutes to complete. After his tip about restarting Docker, I gave it a shot. I have Docker Community Edition 17.06.0-ce-mac19 and I run Sierra (10.12.6). Not sure if that makes any difference.

My docker-sync.yml

version: "2"

options:
  compose-file-path: 'docker-compose.yml'
  compose-dev-file-path: 'docker-compose-osx.yml'
syncs:
  lapis-unison-sync:
    src: '.'
    sync_strategy: 'native_osx'
    sync_excludes: ['.git', '.github', '.idea', '.gitignore', 'node_modules', '.DS_Store']
    sync_userid: ${WWW_DATA_USER_ID}

I give 4 CPUs and 2.0GB to Docker. My host machine has 16GB of memory, and a 2GHz intel Core i5
on the 13" 2016 two thunderbolt 3 version of Mackbook Pro. I can't say for sure how many files I sync, but it's in the early thousands

[chris-aeviator]

I can confirm experiencing the same issue similar to https://github.com/EugenMayer/docker-sync/issues/410#issuecomment-325578867 . My sync is dying mostly multiple times per day, sometimes not at all.

I am syncing a Drupal 8 environment with roughly 64.000 files with all dependencies - as long as everything is fine - everything is fine :)

I realized that restarting Docker / Docker-Sync helps sometimes but is not a guarantee. Today I was experiencing multiple "hickups" again and couldnt solve the issue by restarting, but strangely after about 5-10 Min the sync happened without doing anything.

To all others experiencing the issue, try to just wait and look into the log docker-sync logs , everytime I had the dying sync it was hundreds of twig cache files (Drupal related) beeing responsible for the delay.

[zedtux]

I'm now facing the same issue: The sync is working and after a while it simply stops.

Looking around in the sync volume container, I can see the files well synchronised on the /host_sync folder, but not in the /app_folder (this is true for file creation/editing/deletion).
The top command shows me the normal processes running, no high load, and inotify (with the alpine image, inotify is not available, so I used inotifyd and inotifywatch, but they are blind).

To solve this, I have to stop the stack, run a docker-sync-stack clean and docker-sync-stack start.

I'm trying to read more about unison in order to check who to get more information from it, but for now I can't figure out why is that happening.

[zedtux]

Short update on this ticket, I'm exploring the -fastcheck flag in order to solve this issue.

Fast Update Detection

If your replicas are large and at least one of them is on a Windows system, you may find that Unison's default method for detecting changes (which involves scanning the full contents of every file on every sync—the only completely safe way to do it under Windows) is too slow. Unison provides a preference fastcheck that, when set to true, causes it to use file creation times as 'pseudo inode numbers' when scanning replicas for updates, instead of reading the full contents of every file.

When fastcheck is set to no, Unison will perform slow checking—re-scanning the contents of each file on each synchronization—on all replicas. When fastcheck is set to default (which, naturally, is the default), Unison will use fast checks on Unix replicas and slow checks on Windows replicas.

This strategy may cause Unison to miss propagating an update if the modification time and length of the file are both unchanged by the update. However, Unison will never overwrite such an update with a change from the other replica, since it always does a safe check for updates just before propagating a change. Thus, it is reasonable to use this switch most of the time and occasionally run Unison once with fastcheck set to no, if you are worried that Unison may have overlooked an update.

Fastcheck is (always) automatically disabled for files with extension .xls or .mpp, to prevent Unison from being confused by the habits of certain programs (Excel, in particular) of updating files without changing their modification times.

I updated my docker-sync.yml file by adding sync_args: '-fastcheck no', then I did a docker-sync-stack clean and then docker-sync-stack start.
Since yesterday afternoon is working so I'm continuing with this config and will update this ticket if any issue occurs again.

[zedtux]

I worked the whole morning with this setup without any sync issue anymore. Looks promising! 😎

[EugenMayer]

@zedtux i do not expect this to work, since our issue is rather not unison and its detection, much more it is the OSXFS event layer stalling - and fastcheck will not help here at all, since it tries to make unison faster ( and probably it will ) - but without fs events, unison does nothing - and that is what i strongly expect to happen in those cases.

If it would be unison, restarting the sync container ( unison ) would help - it does not. Restarting the stack would help too ( obviously ) - it does not. Restarting d4m does -> restarting hyperkit -> resetting OSXFS

[zedtux]

fastcheck will not help here at all, since it tries to make unison faster ( and probably it will )

Well actually I'm slowing down it by setting it to no.

Okay, maybe I'm wrong, but anyway, in my case I had sync issue after a while, and thanks to this, it is now working fine.

Sorry for the noise.

[EugenMayer]

@zedtux no sorry needed, i am glad you try to fiddle this out - and mine are assumptions on stuff we did find out - but my opinion is far from being a fact. So thinking in all direction is great help - by no means its noice - its rather trying and at very least, removing one possibility from the equation or even find the solution

[michaelbaudino]

That's actually a good idea to try @zedtux 👍 But like @EugenMayer I don't think it'll work, because (in my understanding!) what -fastcheck no does is to use file content (instead of file modification time) to determine if the file has changed. But (as far as I understand, again!), those checks (whatever method is used) are triggered by filesystem events… and those filesystem events not being passed through by OSXFS is the root of our problem. Thus, whatever check detection we configure, I'm afraid it won't be used anyway 😢

But that's absolutely no noise, and indeed worth a try 🤗

In the same kind of thinking, maybe unison-fsmonitor has an option to check the whole tree every N seconds rather than relying on FS events? If such a setup prevents this issue, it would confirm our hypothesis again and would definitely narrow down the scope of investigation (I didn't find any documentation about such an option, unfortunately, and lost myself in the Unison Makefiles and OCaml code when trying to figure out how it was working… src/fsmonitor/linux/watcher.ml? src/fswatch.ml? src/unison-fsmonitor.py? 😱)

[mike-es]

I have a few notes that support the suspicion of OSXFS events stalling. I am working on a PHP app using Laravel

• files created in php container sync to host
• files created on host sync to /host_sync but unison fails to sync to /app_sync with no logs
• restarting unison causes files to sync (supervisorctl restart unison)
• all sync containers break at the same time (3 total)
• time/tz are the same between host and sync containers
• files created directly in /host_sync sync to /app_sync
• it seems like sync breaks when laravel throws an error (logs are excluded)

Docker version:
Docker version 17.09.0-ce, build afdb6d4

Docker sync version:
0.4.6

macOS version:
10.12.6 (16G29)

[zedtux]

Thank you guys for your comments. Honestly, since I set this flag, all my files are well sync, full stop.

Maybe give it a try, and see if it solves the issue. If it does, then we can try to tacle this issue from a new angle.

[DavidDeSimone]

My team and I are also experiencing this issue. I am able to reproduce it pretty easily by switching around our repo to older changesets. I'm seeing it with a changeset that effects around 15,000 files. The behavior I was seeing was consistent with what other's have commented, inotifywatch/inotifywait were not seeing events, restarting d4m fixes the issue.

Since I am able to repro this, I will try to verify @zedtux suggestion.

I am attempting to create a sample project that can isolate this issue. We will be able to get help from the docker team easier if we can give them a project that can easily show off what is happening. It seems to be difficult to trigger for small data changes (> 200M). I might just clone a large repo like emacs and jump around that to trigger the behavior in a way that will be accessible to everyone.

[murnieza]

Our team is also heavy suffering from this issue. We even dropped docker for now and switched to Vagrant env.

[mike-es]

I tried @zedtux suggestion by adding sync_args: '-fastcheck no' to 2/3 sync configs. I still run into this issue on all 3 sync containers.

[DavidDeSimone]

Based on my testing, it seems as if there is a building queue of inotify events that either isn't being flushed, or is very slow to flush. If I rapidly switch branches, I will no longer see events on a monitored file if I edit it from my mac, however I always see changes instantly if I update from within the container. The timeline is:

• Rapidly change a large number of files (15000+)
• Edit a file that is not in the changeset from OSX. (no inotify)
• Edit that same file from within the container (inotify)

Sometimes if I wait long enough, all of the "pending" inotify events seem to happen at once, however this doesn't always happen.

[zedtux]

@mike-es did you run the docker-sync-stack clean? What do you do which makes the sync no more working? Just working normally? Switching branches? Some thing else?

[mike-es]

I'm using docker-sync start|clean but yes I did clean. I caused the issue by switching branches on the repo without -fastcheck no which I would expect to break that sync but not the others. However, all of them broke.

Is that configuration still working for you?

[mike-es]

@zedtux I just ran another test. I ran docker-sync clean and added sync_args: '-fastcheck no' to all sync configs. When I brought up my stack syncing was broken right off the bat. I have not restarted d4m today.

[mike-es]

I ran another test inspired by docker/for-mac#681. I wanted to see if inotify events broke for other containers that have nothing to do with docker-sync.

Dockerfile:

FROM alpine

RUN apk --no-cache add inotify-tools
COPY wait.sh /bin/

Wait script (wait.sh):

#!/bin/sh
FORMAT=$(echo -e "\033[1;33m%w%f\033[0m written")
"$@"
while inotifywait -q --format "$FORMAT" /mnt
do
  "$@"
done

Run command (672463996aa6 is image build from Dockerfile above):

docker run --rm -it -v /path/to/UnrelatedProject:/mnt 672463996aa6 /bin/wait.sh

Test 1: I switched branches multiple times in UnrelatedProject which is a fairly large code base. Each time I switched I got inotify events from the wait script. No issues.

Test 2: I started up my stack using docker-sync start and docker-compose up. I created a file in UnrelatedProject and saw inotify events. No issues.

Test 3: I switched branches in 1 (out of three) synced repos while watching docker-sync logs -f. The files synced correctly. Then I created a file in UnrelatedProject and saw the inotify event. I repeated this a few more times and slowly started losing inotify events from UnrelatedProject until eventually all events stopped and docker-sync stopped working.

Test 4: I brought down my stack (docker-compose down and docker-sync clean) and I inotify was still broken for UnrelatedProject.

Conclusion 1: This issue has nothing to do with docker-sync and is purely a d4m issue, possibly related to file descriptors since I'm syncing 3 different repos and switching branches.

Conclusion 2: Based on Test 1 causing no issues, the docker-sync unison image is doing something funny that is interfering with the FSEvents/inotify of d4m as a whole. I don't have a lot of knowledge in this area but this seems unlikely.

**UPDATE**

I was able to reproduce the issue by starting 4 of the above containers and not touching docker-sync at all. I bind mounted the 3 repos in my stack and 1 for UnrelatedProject. After switching branches in a few of the repos the inotify events slowed then stopped. This points to a d4m issue and not docker-sync.

Unfortunately this is a major blocker for my team using Docker at all. I switched to the rsync strategy which is working pretty well, but still running into a few issues.

[zedtux]

Is that configuration still working for you?

@mike-es yes it is.

When I brought up my stack syncing was broken right off the bat.

I never faced something like this.

@mike-es what version of d4m are you using?

I tried to change branches and so on but all is working fine for me 😕

[EugenMayer]

@mike-es impressive deduction and system work here. So know with https://github.com/EugenMayer/docker-sync/issues/410#issuecomment-337402855 we can take the suggestion we had https://github.com/EugenMayer/docker-sync/issues/410#issuecomment-316604056 and https://github.com/EugenMayer/docker-sync/issues/410#issuecomment-316614966 is a fact finally.

its fsevents related and since you can force it using a higher amount of containers what it tells us is, the amount of events is the key factor, correlating to the amount of files "changeable" - this explains why some of us have no issues or very rare and some of use have more of those. It depends on how many files you change at one go. So gulp/grunt/webpack stacks will have more problems then ruby/PHP/java projects.

@mike-es and all thank you a lot on participating on this issue and helping to iron out the core of the problem.

I would say, we probably know, using the isolated description of @mike-es create a d4m issue to see if they can give us more insight or even a fix or even something how to flush the event queue without restarting the stack. What do you guys think?

[EugenMayer]

Oh @mike-es i have see you already joined a existing issue on that - we had that on our cause list already.

@all please join that issue, bump up using your thumb up to get traction to that issue please:
https://github.com/docker/for-mac/issues/681

[zedtux]

Just to inform you that after more than a day without any issues, I added the flag to all the other projects (in the same docker-sync.yml file), I did clean everything and started again the stack but I'm now facing again the sync issue 😕

[EugenMayer]

@MichaelHindley instead of ranting right away, you might rather starting reading the thread - this is disrespectful you you are pleased to use something else.

docker-sync is not broken, we even cannot do anything about that here at all.

If you use d4m straight, no docker-sync involved, mount in a volume and run a watch task in your container, then start changing code - after some time, as outline here, OSXFS will struggle with FS events and your watch task wont work anymore. No docker-sync involved and zero you can do about this.

But why i am even repeating this at all, its just the same story again, you did not even dare to read this thread or understand, you are just here to dump your rant.

Beside that, thousands of us using docker-sync on a daily base - and yes, once a day this can happen. Inconvinient, yes, steals you 5 minutes restarting d4m and the stack, yes.

but you lose those 5 minutes every hour without using docker-sync waiting for your random framework to evne load up

[EugenMayer]

@MichaelHindley we do not "advertise" - thats not a commercial product. Thats not a bold claim to fool you in and take your money. People in here have 0 benefits from fooling you, lieng to you or gaining anything from false claims.

You are talking about being defensive, while you go straight in swinging the axt.

Please, feel free to make any suggestion were you have spend time and effort to think through it anytime - other then that -if you do not like docker-sync because "it just does not work and all the other stupid people got fooled and just do not get it" ..

please pick a better tool working for you and leave. Not going to comment on this further, just dragging it offtopic

[mmrko]

Meanwhile, the rsync strategy works beautifully. Kudos to @EugenMayer & co. for their hard (and gratuitous) work! :)

[EugenMayer]

@mmrko yes, rsync is not affected by this, thats why having multiple strategies is one of our goals - pick the axe for your case. The problem with rsync is on the other side, CPU load, since it uses fswatcher to watch for file changes - and due to macOS being horrible in FS-Events, even that is just bad as it is. So its pretty bullet proof, but not entirely.

But more or less, all strategies have their little edges, thats assured. But running drush for streight up 2 minutes for a drush cc all on the cli alone, instead of 3 seconds - is a much harder "edge" for most of us, i guess.

I would love to remove docker-sync from this world, if it would not be needed .e.g d4m fixed it. That will be a great day for us all - no overhead, no glitches, no extra configuration - just docker.

Well d4m is not just docker, its docker on a non linux kernel...and thats where we all start.

[EugenMayer]

Closed the thread for collaborators only so this thing does not get offtopic even more - way to important issue with way to valuable information, no need for all the other.

Will reopen the issue in 1 week when things calmed down. If you feel in the need to express yourself about this, open a new issue in the queue so this one keeps the focus

[EugenMayer]

Anybody having this issue, please see https://github.com/EugenMayer/docker-sync/issues/497 and run the reproduction script, post your details.

Also ensure you test it 2 times, one without my d4m fs fix and one with: https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2

Does the result change for you? This could really help use nailing down the issue.

Why i am assuming that my fix helps is, that neither i have this issue often ( about 1 time in 3 months ) neither anybody of my coworkers - we are doing heavy development though, it means, one project most probably having 0.5 million files, 5 docker-sync endpoints starting at the same time and a stack which consume 8GB Ram + quiet some CPU. Still it hardly ever happens for us ( i think for most of us, it never happened at all). We have other projects, bigger Java projects, bigger ng4 stacks and so on, still it hardly ever happens for us.

So there must be something we do that this does not happen. One thing i can tell is:

• we are either using Sierra or Mountain Lion
• we are all using that fix above
• we are all having about 8-16GB RAM assigned to d4m
• at least 2-4 CPUs ( depending if it is a desktop or laptop )

[EugenMayer]

i am currently thinking about closing this issue and open a new one, extracting all the facts / informations we had here, also how to test owns system to provide informations so we can see pattern. I would link this issues as the source of deeper discussions.

Any concerns about that? Main reason is, to consolidate the results and make it easier to understand how this issues shows up, how you can for now work around and what the reasons for this are, without reading so many posts.

[zedtux]

@EugenMayer I agree with you, it should also allow new comers to quicker understand the issue and where we stand.

[ernestas-s]

And I think we should have in mind, what projects we are doing. Because I'm working on symfony project, and I only sometimes have conflicts and had this no sync problem only maybe twice in 5 months. But frontend team working with same config, having this problem few times a week. They using Angular 1/2.

version: "2"
syncs:
  backend-sync:
    src: '../backend'
    sync_strategy: 'unison'
    sync_excludes: ['.git', '.idea']
    # sync_excludes: ['.git', '.idea', 'var/cache']
    sync_excludes_type: 'Path'
    watch_strategy: 'fswatch'

  frontend-sync:
    src: '../frontend'
    sync_strategy: 'unison'
    sync_excludes: ['.git', '.idea']
    sync_excludes_type: 'Path'
    watch_strategy: 'fswatch'

[b1alpha]

Is this fixed? A few devs are starting to see docker-sync just stop. logs dont show any issue its just not picking up any changes. clean and start didnt seem to resolve then about 5min later it started working again

[ddanier]

@EugenMayer Your gist (https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2) seems not to work any more. I get "fatal: not a git repository (or any of the parent directories): .git". Is there an updates version? Thank you!

[ernestas-s]

Our team gave up. We went to bare metal dev env. Some have VMware.

[markshust]

If you use the native delegated volumes, docker-sync isn't needed anymore as far as I'm concerned. I've had solid performance on Magento 2 (TONS of folders/files) with no delay or halts in syncing.

[EugenMayer]

closing this, use d4m edge to get it very robust, we have a about one sync stop per week at max

[EugenMayer]

@ddanier this is no longer needed with d4m edge

[ohcibi]

I just started using docker-sync with latest versions of everything. Initial sync works but then nothing gets sync. Even with explicitly running docker-sync sync. I just saw an error message:

unix:///tmp/supervisor.sock refused connection

When I tried to restart everything as suggested here.

[EugenMayer]

I would suggest opening a ticket, paste your configuration there and continue from there

[adriadrop]

Maybe just to sum this up. Solution for this would be to either put docker4mac edge or use rsync as sync strategy?

[EugenMayer]

d4m will do it (in the meantime)

Or to just understand how to configure your excludes and have a proper project layout mitigating the issues of d4m.

Reduce the amount of files you

• watch
• you sync

to _only_ the parts you edit.

[infostreams]

Maybe I was to stupid and didn't install everything correctly, but I think I followed the installation instructions correctly. However, docker-sync never worked for me.

I mount folder X from my Mac to docker-sync, and I'm running docker-sync with docker-machine. If I changed a file in folder X on my Mac, I would see it in the host_sync/ folder, but unison never picked up the change, and the change _never_ ended up in the app_sync/ folder, and thus _never_ in my own docker container.

However, if I opened a shell in the docker container running docker-sync, and manually made a change in the host_sync/ folder from within the docker-sync container (e.g. by doing touch /host_sync/examplefile.txt), then unison would see that something changed, and copy it to the app_sync/ folder. So, the problem was with Unison, somehow.

After a long search it turns out that this particular behavior is related to a WONT-FIX bug in VirtualBox (see here and here). Basically, a change to a file that was initiated on my Mac does not result in a filesystem event within the container, and is therefore never picked up by Unison. Unison's documentation even explicitly states this (see documentation: "_In particular, the Unix implementation does not compare the actual contents of files to their previous contents, but simply looks at each file's inode number and modtime; if neither of these have changed, then it concludes that the file has not been changed._")

So, I'm now running a separate program inside the docker-sync container that regularly polls my host_sync/ folder, and whenever it detects a change, it touches the file, which causes a filesystem event inside the container, which then causes Unison to pick up the change, which means my changes end up in my own container 🎉

I initially installed inotify-proxy to do this, but I didn't manage to get it to work properly, so I coded up my own version in PHP. It's plenty fast and seems to work as intended. Hope this helps someone here.

If this whole story above is old news and there's existing code in docker-sync to deal with this, can you point me to it, and either update the documentation to point it out clearly, or make it the default behavior? As I interpret the Unison documentation (_"considered unchanged if inode is the same"_) and the VirtualBox documentation (_"we don't pass on inode changes"_) I don't think the existing setup could have ever worked for Docker running on VirtualBox.

[EugenMayer]

@infostreams this and more reasons are, why you should use OSXFS, thus d4m and not vbox/fusion, since the FS driver of OSXFS is better in those terms and fires all the events.

There is no doubt that docker4mac and OSXFS has issues too, but i consider it the better alternative.

Thanks for sharing your story in that detailed fashion though, very appreciated!

[infostreams]

@EugenMayer I don't think the documentation mentions anywhere that this, out of the box, doesn't work with docker-machine. If I had known that I could have resolved my issues by switching to docker4mac, I would have done that. As it is I spent a day or so on this problem. Can you update the documentation to say something along these lines? It would have saved me (and no doubt others) a lot of frustration.

[EugenMayer]

@infostreams checking the docs, i really have no topic about d4m vs docker-machine dedicated to this topic beside the tests under https://docker-sync.readthedocs.io/en/latest/miscellaneous/performance.html

But we had this https://github.com/EugenMayer/docker-sync/issues/346 ..so exactly your finding so indeed we could have saved some time for you.

But that is how docs are, they are never complete.

We have a topic about how to pick the right strategy at https://docker-sync.readthedocs.io/en/latest/advanced/sync-strategies.html and that currently suggest OSXFS eventhough it does not explain all the details, including those details - but following that guide would have saved you time too :)

Anyway, we should probably add a "docker-machine vs d4m/d4w" topic - are you willing to do a PR for that? That would be awesome

Update: it was there: https://docker-sync.readthedocs.io/en/latest/advanced/sync-strategies.html

It works under Docker for Mac only - missing file system events under vbox/fusion. See native_osx does not work with docker-machine vbox / fusion

under cons

[infostreams]

Hi, thanks for pointing out that reference. However, I completely missed that single sentence, and I wouldn't be surprised if many others missed it as well. I will open a pull request to add a single sentence to the installation instructions for Mac, because only saying "With native_osx we no longer have any host dependencies" is slightly misleading since it only works on one of the two ways to run Docker on Mac. Thanks for your great work!

[ostrolucky]

Hey guys, I have run into this issue recently. I've read the thread and it mentions it shouldn't happen with d4m edge anymore. But here we are, with d4m 2.2.3.0 edge and it still happens. Restart of my Mac fixed it. I think I didn't try restarting Docker itself though, could maybe help as well.

[EugenMayer]

@ostrolucky this is known and edge does not fix it ( or anything ). Either FSevents get stuck inside macOS ( macOS restart needs restart ) or inotify events inside the kernel of d4m ( docker restart might help ) and sometimes it's unison inside the sync container getting overheated by the amount of inotify events ( huge file change chunks ) which can be fixed by either restarting unison and sometimes in addition removing the lockfile of unison

To be frank, this is unfixable by docker-sync - this is broken conceptually by macOS ( horrible FSevents implementation ), leading to a half baked OSXFS implementation on top of it, and in the end unison, which simply cannot handle fast changes of 20k files like an npm install would do

[ostrolucky]

this is known and edge does not fix it ( or anything )

Good, this is just to make that transparent, because reading this thread it implies this was fixed in d4m edge. So good to make that clear that it indeed doesn't fix it.

which simply cannot handle fast changes of 20k files

Do you think mutagen can? If it does, perhaps we should write a strategy for it in docker-sync. Because TBH it's annoying having to exclude such folders with dependencies. Although we are a bit OT now, but there is no community forum or anything like that for docker-sync AFAIK.

[EugenMayer]

@ostrolucky there is no community forum and yes, the latter might be fixable by mutagen, but nobody really tried - see #603 for a issue about implementing mutagen. Feel free to join there and maybe even contribute with a PR or try to bring mutagen to those edge cases.

I will close this thread with repeating the answer above so people now the "essence" - got your point, good idea

[EugenMayer]

d4m does not fix it - there are several layers of issues:

• Either FSevents get stuck inside macOS ( macOS restart needs restart ) \
• or inotify events inside the linux kernel of d4m ( docker restart might help )
• and sometimes it's unison inside the sync container getting overheated by the amount of inotify events ( huge file change chunks ) which can be fixed by either restarting unison and sometimes in addition removing the lockfile of unison

To be frank, this is unfixable by docker-sync - this is broken conceptually by macOS ( horrible FSevents implementation ), leading to a half baked OSXFS implementation on top of it, and in the end unison, which simply cannot handle fast changes of 20k files like an npm install would do.

So there are cases where sync stops, many of them fall in the one mentioned above - and we can only try to mitigate those