Docker-sync: Error starting sync, exit code 0

[BKStephens]

This project looks awesome! I am trying to get it running, but am running into some issues.

When I clone this repo, and try to run just the unison-sync example (removed other syncs from docker-sync.yml), I get:

example master % docker-sync start
          ok  Starting unison
          ok  unison-sync container not running
          ok  creating unison-sync container
     command  docker run -p '127.0.0.1::5000'                               -v unison-sync:/data                               -e UNISON_DIR=/data                               -e TZ=${TZ-`readlink /etc/localtime | sed -e 's,/usr/share/zoneinfo/,,'`}                               -e UNISON_OWNER="www-data" -e UNISON_OWNER_UID="33"                                                              --name unison-sync                               -d mickaelperrin/docker-unison-unox
          ok  starting initial sync of unison-sync
     command  docker inspect --format=" {{ .NetworkSettings.Ports }} " unison-sync | /usr/bin/sed  -E "s/.*map\[5000[^ ]+ ([0-9]*)[^0-9].*/\1/"
     command  unison /Users/benstephens/repos/docker-sync/example/data4 -auto -batch socket://localhost:32775
       error  Error starting sync, exit code 0
     message  Contacting server...
Fatal error: Failed to connect to the server on host localhost:32775
     success  Unison server started
     command  docker inspect --format=" {{ .NetworkSettings.Ports }} " unison-sync | /usr/bin/sed  -E "s/.*map\[5000[^ ]+ ([0-9]*)[^0-9].*/\1/"
     command  unison /Users/benstephens/repos/docker-sync/example/data4 -auto -batch socket://localhost:32775
       error  Error starting sync, exit code 0
     message  Contacting server...
Fatal error: Failed to connect to the server on host localhost:32775
     command  docker inspect --format=" {{ .NetworkSettings.Ports }} " unison-sync | /usr/bin/sed  -E "s/.*map\[5000[^ ]+ ([0-9]*)[^0-9].*/\1/"
     command  unison /Users/benstephens/repos/docker-sync/example/data4 -auto -batch socket://localhost:32775 -repeat watch
Contacting server...
Fatal error: Failed to connect to the server on host localhost:32775

Here is some information about my system:
OS X El Capitan Version 10.11.16
Docker version 1.12.0, build 8eab29e
docker-compose version 1.8.0, build f3628c7
Docker for Mac 1.12.0-a
ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-darwin15]
unison: stable 2.48.4 (bottled)

[mickaelperrin]

Thx for reporting.

Could you paste your config and the result of `docker ps -a | grep unison' ?

[EugenMayer]

@BKStephens any reason you are using the development version instead of the current released one? if you just start with the project, i would really recommend you using the current release

gem install docker-sync

instead of the current development master

if you want to test-drive the project, use the boilerplate at https://github.com/EugenMayer/docker-sync-boilerplate rather then the examples in the master, they are less maintained right now. Thanks!

@mickaelperrin i would rather not wonder if the examples are right now broken.

[BKStephens]

For config, everything is the same as https://github.com/EugenMayer/docker-sync/tree/master/example, except for docker-sync.yml. For docker-sync.yml, I removed syncs besides unison:

# for a simpler example, see the boilerplate at https://github.com/EugenMayer/docker-sync-boilerplate
options:
  # optional, default: docker-compose.yml if you like, you can set a custom location (path) of your compose file
  compose-file-path: 'docker-compose.yml'
  # optional, default: docker-compose-dev.yml if you like, you can set a custom location (path) of your compose file. Do not set it, if you do not want to use it at all
  # if its there, it gets used, if you name it explicitly, it HAS to exist
  compose-dev-file-path: 'docker-compose-dev.yml'
  # optional, activate this if you need to debug something, default is false
  verbose: true
  # ADVANVED: default:eugenmayer/rsync -  the image to use for the rsync container. Do not change this until you exactly know, what you are doing
  # you cannot just use "some unison" container, you entrypoint needs features, check the Dockerfiles
  rsync_image: 'eugenmayer/rsync'
  # ADVANVED: default:eugenmayer/unison -  the image to use for the unison container. Do not change this until you exactly know, what you are doing
  # you cannot just use "some unison" container, you entrypoint needs features, check the Dockerfiles
  unison_image: 'eugenmayer/unison'
  # can be docker-sync, thor or auto and defines, how the sync will be invoked on the cli. Mostly depending if your are using docker-sync solo, scaffolded or in development ( thor )
  cli_mode: 'auto'
syncs:
  unison-sync:
      src: './data4/'
      dest: '/data'
      image: 'mickaelperrin/docker-unison-unox'
      sync_host_ip: 'localhost'
      sync_strategy: 'unison'
      sync_user: 'www-data'
      sync_userid: '33'
      # If you need to sync a lot of files, you can reach out the system limit
      # of inotify watches. You can set the limit by using this parameter. This will
      # prompt you for your sudo password to modify the system configuration.
      #max_inotify_watches: 100000

Is that what you are looking for?

After opening this issue, I stopped Docker for Mac and started docker-machine again to go back to my old setup for the time being. After seeing your response, I stopped docker-machine and started Docker for Mac, and am seeing different results. This is surprising as I saw the output I posted above last night, then again today after restarting my computer.

Anyway, it looks like it is making it further:

example master % docker-sync start
          ok  Starting unison
          ok  unison-sync container not running
          ok  creating unison-sync container
     command  docker run -p '127.0.0.1::5000'                               -v unison-sync:/data                               -e UNISON_DIR=/data                               -e TZ=${TZ-`readlink /etc/localtime | sed -e 's,/usr/share/zoneinfo/,,'`}                               -e UNISON_OWNER="www-data" -e UNISON_OWNER_UID="33"                                                              --name unison-sync                               -d mickaelperrin/docker-unison-unox
          ok  starting initial sync of unison-sync
     command  docker inspect --format=" {{ .NetworkSettings.Ports }} " unison-sync | /usr/bin/sed  -E "s/.*map\[5000[^ ]+ ([0-9]*)[^0-9].*/\1/"
     command  unison /Users/benstephens/repos/docker-sync/example/data4 -auto -batch socket://localhost:32770
          ok  Synced /Users/benstephens/repos/docker-sync/example/data4
      output  Warning: No archive files were found for these roots, whose canonical names are:
        /Users/benstephens/repos/docker-sync/example/data4
        //a2d0152e9b0b//data
This can happen either
because this is the first time you have synchronized these roots,
or because you have upgraded Unison to a new version with a different
archive format.

Update detection may take a while on this run if the replicas are
large.

Unison will assume that the 'last synchronized state' of both replicas
was completely empty.  This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.

If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations.  See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.

Donations to the Unison project are gratefully accepted:
http://www.cis.upenn.edu/~bcpierce/unison

     success  Unison server started
     command  docker inspect --format=" {{ .NetworkSettings.Ports }} " unison-sync | /usr/bin/sed  -E "s/.*map\[5000[^ ]+ ([0-9]*)[^0-9].*/\1/"
     command  unison /Users/benstephens/repos/docker-sync/example/data4 -auto -batch socket://localhost:32770
          ok  Synced /Users/benstephens/repos/docker-sync/example/data4
      output
     command  docker inspect --format=" {{ .NetworkSettings.Ports }} " unison-sync | /usr/bin/sed  -E "s/.*map\[5000[^ ]+ ([0-9]*)[^0-9].*/\1/"
     command  unison /Users/benstephens/repos/docker-sync/example/data4 -auto -batch socket://localhost:32770 -repeat watch
Contacting server...
Connected [//Bens-MBP.home//Users/benstephens/repos/docker-sync/example/data4 -> //a2d0152e9b0b//data]
Looking for changes
Traceback (most recent call last):
  File "/usr/local/bin/unison-fsmonitor", line 24, in <module>
    import fsevents
ImportError: No module named fsevents
Fatal error: File monitoring helper program not running

Here is the result of docker ps -a | grep unison:

example master % docker ps -a | grep unison
a2d0152e9b0b        mickaelperrin/docker-unison-unox   "/entrypoint.sh super"    About a minute ago   Up About a minute                 127.0.0.1:32770->5000/tcp   unison-sync
39b6d00f855c        alpine                             "watch -n1 'cat /app/"    2 minutes ago        Exited (137) About a minute ago                               unison
7446229b3601        alpine                             "watch -n1 'cat /app/"    2 minutes ago        Exited (137) About a minute ago                               unisononsidedapp
edec6fbeaa29        alpine                             "cat /app/cat/somefil"    14 minutes ago       Exited (0) 14 minutes ago                                     example_unison_run_3
4674199fc51c        alpine                             "ls"                      14 minutes ago       Exited (0) 14 minutes ago                                     example_unison_run_2
1cbf24e8a309        alpine                             "bash"                    14 minutes ago       Created                                                       example_unison_run_1
0ce116b83fff        eugenmayer/unison:unox             "/entrypoint.sh super"    15 hours ago         Exited (0) 15 hours ago                                       engagement-data-sync

[EugenMayer]

@mickaelperrin as expected, the example is outdated.

image: 'mickaelperrin/docker-unison-unox' is the issue @BKStephens

please use the boilerplate https://github.com/EugenMayer/docker-sync-boilerplate/tree/master/unison to testdrive docker-sync - sorry for the confusion

rm the image docker rmi mickaelperrin/docker-unison-unox so things do not get confused, the stable release (right now) is eugenmayer/unison:unox, which is based on mickaelperrin image, but is probably a newer build

[BKStephens]

I did not intentionally install a development version of the gem. I used gem install docker-sync to install. Why do you think it is the development version?

OK, that is good to know that https://github.com/EugenMayer/docker-sync-boilerplate has up-to-date examples.

For https://github.com/EugenMayer/docker-sync-boilerplate/tree/master/unison , it looks like I am getting a similar error as reported in https://github.com/EugenMayer/docker-sync/issues/119#issuecomment-241063727:

unison master % docker-sync start
          ok  Starting unison
          ok  Synced /Users/benstephens/repos/docker-sync-boilerplate/unison/app
     success  Unison server started
          ok  Synced /Users/benstephens/repos/docker-sync-boilerplate/unison/app
Contacting server...
Connected [//731a53147bc4//var/www -> //Bens-MBP.home//Users/benstephens/repos/docker-sync-boilerplate/unison/app]
Looking for changes
Traceback (most recent call last):
  File "/usr/local/bin/unison-fsmonitor", line 24, in <module>
    import fsevents
ImportError: No module named fsevents
Fatal error: File monitoring helper program not running

[EugenMayer]

it seems like pip install macfsevents , as a companion to unison we use unox for even-watching, and thsi depends on fsevents.

what happens when you do
where python and where pip

what happens when you write pip install macfsevents does it fail due to permission issues ( do not use sudo for now please )

[EugenMayer]

@BKStephens join me on gitter https://gitter.im/EugenMayer/docker-sync we probably can deal with it easier there and post the result here

[EugenMayer]

this is related or lets say a "dupe" of https://github.com/EugenMayer/docker-sync/issues/112 and was cause by pip install macfsevents which is needed for unison-fsmonitor, not to work, since the system python requires sudo

@BKStephens thank you for taking your time to track this down!

[BKStephens]

Fixed by dfde2d718b6078b5f7e8e12c4245f13212d7fdd6.

Thank you for all of your help and for your work on this project!

For the sake of history, here is our Gitter conversation on the issue:

Ben Stephens @BKStephens 09:37
Hello @EugenMayer. In regards to https://github.com/EugenMayer/docker-sync/issues/119#issuecomment-241067218, where pip returns pip not found

Eugen Mayer @EugenMayer 09:37
@mickaelperrin i am moving to a new house the coming week an probably wil have not a single second this weeken, but i try!
hi @BKStephens. ok, what does where python tell you and what where easy_install

Ben Stephens @BKStephens 09:39
unison master % where python
/usr/bin/python
unison master % where easy_install
/usr/bin/easy_install

Eugen Mayer @EugenMayer 09:39
interesting ben, do me a favour, do the following
gem remove docker-sync
rm ~/.docker-sync-global.,yml
then install it once again gem install docker-sync
@mickaelperrin i think https://github.com/EugenMayer/docker-sync/blob/master/lib/docker-sync/preconditions.rb#L48 is kinda failing

Ben Stephens @BKStephens 09:41
unison master % gem remove docker-sync
ERROR:  While executing gem ... (Gem::CommandLineError)
    Unknown command remove
unison master % gem uninstall docker-sync
Remove executables:
        docker-sync, docker-sync-stack

in addition to the gem? [Yn]  Y
Removing docker-sync
Removing docker-sync-stack
Successfully uninstalled docker-sync-0.1.1
unison master % rm ~/.docker-sync-global.yml
unison master % gem install docker-sync
Fetching: docker-sync-0.1.1.gem (100%)
Successfully installed docker-sync-0.1.1
1 gem installed

Eugen Mayer @EugenMayer 09:41
@mickaelperrin or lets say, the binary is there, but without the module, its useless. And when the binary has been install 'correctly' there will be no additional try to install it
@BKStephens great, now we have to uninstall unox-monitor
@BKStephens rm /usr/local/bin/unison-fsmonitor

Ben Stephens @BKStephens 09:42
done
Now try docker-sync start again?

Eugen Mayer @EugenMayer 09:43
ok Ben, now go to the boilerplate/unison folder and start docker-sync .. it should tell you, that /usr/local/bin/unison-fsmonitor is missing and asks you to install it for you, say yes
@BKStephens exactly
please, after it "installed" it ( it actually fails but probably does not say it to loud ) paste your console-output here or on pastebin / gist /similar
@BKStephens thank you for your time . Its the usual "how to install libs on OSX madness" 

Ben Stephens @BKStephens 09:44
unison master % docker-sync start
     warning  Please be aware that with the strategy "unison" is now called unison-onesided and you might need to migrate. See https://github.com/EugenMayer/docker-sync/wiki/Migration-Guide for more informations
Shall we continue? Y
     warning  Could not find unison-fsmonitor (for file watching) binary in $PATH. Please install unox before you continue, see https://github.com/hnsl/unox.
Shall I install unison-fsmonitor for you? Y
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9720  100  9720    0     0   5369      0  0:00:01  0:00:01 --:--:--  5370
          ok  install macfsevents using pip
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/test-easy-install-5990.pth'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /Library/Python/2.7/site-packages/

Perhaps your account does not have write access to this directory?  If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account.  If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

  https://pythonhosted.org/setuptools/easy_install.html

Please make the appropriate changes for your system and try again.

          ok  Starting unison
docker: Error response from daemon: Conflict. The name "/appcode-unison-sync" is already in use by container 9530f4a457377a1cd05a17a8bfbd95717952ba4f12d2851b6a3e82edb4764c50. You have to remove (or rename) that container to be able to reuse that name..
See 'docker run --help'.
          ok  Synced /Users/benstephens/repos/docker-sync-boilerplate/unison/app
     success  Unison server started
          ok  Synced /Users/benstephens/repos/docker-sync-boilerplate/unison/app
Contacting server...
Connected [//9530f4a45737//var/www -> //Bens-MBP.home//Users/benstephens/repos/docker-sync-boilerplate/unison/app]
Looking for changes
Traceback (most recent call last):
  File "/usr/local/bin/unison-fsmonitor", line 24, in <module>
    import fsevents
ImportError: No module named fsevents
Fatal error: File monitoring helper program not running
Np, thank you for all the support @EugenMayer !

Eugen Mayer @EugenMayer 09:45
ok we got it!
@mickaelperrin i was right with sudo:
[Errno 13] Permission denied: '/Library/Python/2.7/site-packages/test-easy-install-5990.pth'
ok @BKStephens stop the process

Mickaël PERRIN @mickaelperrin 09:46
Ok, so the problem is the missing sudo or there is also anything else ?

Eugen Mayer @EugenMayer 09:47
@mickaelperrin thats what we will see, let me try that

Ben Stephens @BKStephens 09:47
docker-sync start stopped

Eugen Mayer @EugenMayer 09:47
@BKStephens docker-sync clean && gem remove docker-sync && rm /usr/local/bin/unison-fsmonitor
@mickaelperrin its this #112
i just try to fix it. The reason @mickaelperrin we did not need sudo is, that i use boxen and thus a user-space python, not the system python..

Ben Stephens @BKStephens 09:48
unison master % docker-sync clean && gem remove docker-sync && rm /usr/local/bin/unison-fsmonitor
     success  Finished cleanup. Removed stopped, removed sync container and removed there volumes
ERROR:  While executing gem ... (Gem::CommandLineError)
    Unknown command remove
unison master % docker-sync clean && gem uninstall docker-sync && rm /usr/local/bin/unison-fsmonitor
Error response from daemon: No such container: appcode-unison-sync
Error response from daemon: No such container: appcode-unison-sync
Error response from daemon: get appcode-unison-sync: no such volume
     success  Finished cleanup. Removed stopped, removed sync container and removed there volumes
Remove executables:
        docker-sync, docker-sync-stack

in addition to the gem? [Yn]  Y
Removing docker-sync
Removing docker-sync-stack
Successfully uninstalled docker-sync-0.1.1

Eugen Mayer @EugenMayer 09:49
@BKStephens go into the git folder for docker-sync wait for me a sec, i do a commit
@BKStephens git pull and then
./deploy_locally.sh
this will install the "master" version of docker-sync as a gem

Ben Stephens @BKStephens 09:52
done

Eugen Mayer @EugenMayer 09:52
rm ~/.docker-sync-global.yml and then go the the unison folder and docker-sync start - you will asked to install, and then to login (sudo) and then you should be good to go - hopefuly

Ben Stephens @BKStephens 09:57
I think that did it. Thanks!

[EugenMayer]

Thanks and happy development :)

[ssbrewster]

Is using sudo pip install the right route to go down here? There are well documented issues with doing that (but I could be wrong in this case). I still got the error anyway even after sorting out permissions. What fixed it for me was cleaning up brew doctor warnings specifically: removing another python version from my PATH and linking up a python keg that was unlinked (I didn't test docker-sync start in between these two).

[EugenMayer]

@ssbrewster the answer is rather complex then simple.

Homebrew
If you use homebrew, you should normally not need sudo, since your pip will be the homebrew python, _if_ any installed there and therefore will be installed in the user controlled folders.
So it will not use the system pip / python, but the overlaying one.

The issue with that so is, that properly configuring your shell with homebrew to user the overlay and also do not get confused by pyenv and things like that - that is were stuff gets complicated.
So you might still be using system python "sometime, depending on the shell or the folder you are in" eventhough you have homebrew

System Python
I can confirm, that it is required to use sudo when using system python pip install since it requires to write into folders controlled by root - this is the same for all systems i have seen running El Capitan and will most probably hold for Sierra.

Does this answer your question?

[ssbrewster]

@EugenMayer thanks yes that answers it and good point on System Python.