Docker-sync: Troubleshooting docker sync on high sierra

[rodrigoaguilera]

Error

After around 10-20 simple edits of files in the host docker sync stops syncing from the host to the container. If I edit files inside the container the sync works.
It seems that somehow it stops listening.
I tried text editors and command line utils to change files.

Docker Driver

d4m Docker Community Edition 17.12.0-ce-rc3-mac43 2017-12-15 (Edge)

Sync strategy

native_osx

your docker-sync.yml

version: "2"

syncs:
  # IMPORTANT: this name must be unique and should NOT match your real application container name!
  myapp-sync:
    src: './'
    sync_userid: '33'

OS

OSX 10.13.1 (updating now to 10.13.2)

More info

docker-sync 0.5.2

We have 3 mac systems using docker-sync successfully with the exact same app even on edge.
The difference is that this one uses High Sierra with APFS.

I look at the unison logs and it seems like no notifications for filesystem changes are happening
I am just looking for a way to troublehoot this further. Any hints?

PD: we use edge because of this one:
https://github.com/docker/for-mac/issues/668#issuecomment-351002066

[EugenMayer]

does it work better with stable? or the other way around, is this a AFPS issue or an edge to issue

[nickdani]

I'm facing the same problem after update to 0.5.2. Before the update, it had been working perfectly, after update to 0.5.2 the sync stopped working at all

Docker
Docker version 17.06.0-ce, build 02c1d87

Docker compose
docker-compose version 1.14.0, build c7bdf9e

Docker sync
0.5.2

Mac os
Sierra 10.12.4

Sync strategy
native_osx

There are no errors in docker sync logs

any suggestions?

[rodrigoaguilera]

@nickdani I am using your versions exactly and I didn't experience any problems.

Can you confirm that the sync works for some modifications after you run docker-sync-stack start and then it stops in only one of the directions (edits on host are not reflected). Just to know if we are talking about the same issues.

In your case I would try a complete docker-compose rm && docker-sync clean (in my case it didn't help)

@EugenMayer I will only have access to that laptop with high sierra after the 10th of January

[nickdani]

@rodrigoaguilera

In your case I would try a complete docker-compose rm && docker-sync clean (in my case it didn't help)

didn't help

actually in my case sync does not work at all :(

[hsuabina]

I'm experiencing the same issue. I always run docker-sync with the --foreground flag, and it stops loging any sync task after some time. It does work for a few syncs but then it stops working.

Docker
Docker version 17.12.0-ce-rc3 (edge)

Docker compose
docker-compose version 1.18.0-rc2

Docker sync version
0.5.2

Mac OS
Sierra 10.12.6

Sync strategy
native_osx

docker-sync.yml

version: '2'

options:
  verbose: true

syncs:
  webapp-sync:
    src: './'
    # native_osx as the sync strategy
    sync_strategy: 'native_osx'
    # this will assign ownership and group of the synced files to 'www-data' in webapp container
    sync_userid: '33'
    sync_groupid: '33'
    # this should exclude some files from being synced at all, increasing perfomance
    sync_args:
      - "-ignore='Path .idea'"              # no need to send PHPStorm config to container
      - "-ignore='Path .git'"               # ignore the main .git repo control files
      - "-ignore='BelowPath .git'"          # also ignore .git repos in subfolders such as in composer vendor dirs
      - "-ignore='BelowPath app/cache/*'"   # don't sync cache
      - "-ignore='BelowPath app/logs/*'"    # don't sync logs
      - "-ignore='Path .docker-sync'"       # don't sync this
      - "-ignore='Name .gitignore'"
      - "-ignore='Name .DS_Store'"

[jonaswouters]

Experienced same issues as the reporter. I've downgraded a version and for now, it seems more stable.

[rodrigoaguilera]

@hectorsabina @jonaswouters Can you confirm that the sync stops only on one direction? (host -> container hopefully)

You can do something like

docker exec mycointainer touch /path/to/be/sync/testfile

And see if testfile appears on your host and your unison logs

[jonaswouters]

I'll try this when it occurs again. For now it seems stable by removing 'logs' from the exclude list. Not sure yet what exactly is causing this, but when it starts failing, the container no longer responds.

[hsuabina]

@rodrigoaguilera After a quick test, I confirmed it does only stop in one direction.

container => host sync works fine.

[EugenMayer]

closing for now, reopen if there is anything to follow up

[hsuabina]

@jonaswouters what version did you downgrade to? I downgraded to v0.5.1 and it doesn't seem to help at all (I think docker-sync was previously working for me at this version, so not sure what's going on).

[jonaswouters]

@hectorsabina yeah v0.5.1 does not help. I have to recreate the containers every time I stop them to keep it working. I still need to verify the 2 way problem, but I don't want to intentionally break my setup when I'm trying to be productive :)

[incrize]

I had a same issue. The problem is solved after moving to stable docker version.

[jonaswouters]

Can confirm container => host sync works fine.

[jonaswouters]

@incrize switched back to stable and same problem. Host -> container stops syncing

[rodrigoaguilera]

I think people are reporting the same issue at
https://github.com/EugenMayer/docker-sync/issues/517

Should we unify at least?

[EugenMayer]

i can close this issue if you like, no problem. But as (ranted) a bit on the other issue, i cannot see a pattern, because there is none. Its a pattern of disagreement, lack of information and probably effort. Tell me if you we should move over, i follow your advice

[mnabialek]

I have exactly same problem. For example I restart Docker, then run docker-sync-stack clean and then docker-sync-stacks start, everything is synced but then I run gulp in my host and file created by host is not visible in container. Synchronization from container to host seem to be working without any problem.

[incrize]

I'm currently using docker version:

Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:22:25 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:28:28 2017
 OS/Arch:      linux/amd64
 Experimental: true

It has been working well so far.

Earlier I did a little bit of research before switching to stable version. It turns out, that after some time file system event notifications stopped dispatching into container. In so doing, the files were changed in host_sync, but because fs event wasn't fired, unison sync didn't start.

In my case this problem tends to come up after multiple files change, for example, after switching to old git feature branch.

I think that the problem is with docker osxfs

[EugenMayer]

This is nothing new, only probably earlier version by version, see https://github.com/EugenMayer/docker-sync/issues/410 - and yes, that is FS events related.

Currently i smell that TRIM operations on APFS based hosts do even do worse on this. We cannot do anything about that at all, see the lengthly discussion there

[EugenMayer]

please downgrade to 17.09.0-ce-mac35 stable see the other thread, closing this one, please continue over there