Docker-sync: Unison strategy can't start sync container

[jeizsm]

Docker-sync 0.5.3 and 0.5.4 with unison strategy can't start sync container

Error/Feature Requestion/Docs

       note:  You can also run docker-sync in the background with docker-sync start
          ok  Starting unison for sync relibc-sync
          ok  relibc-sync container not running
          ok  creating relibc-sync container
     command  docker run -p '127.0.0.1::5000' -v relibc-sync:/app_sync -e VOLUME=/app_sync -e TZ=${TZ-`readlink /etc/localtime | sed -e 's,/usr/share/zoneinfo/,,'`} -e UNISON_EXCLUDES="-ignore='Name target'"  --name relibc-sync -d eugenmayer/unison:0.3
          ok  starting initial sync of relibc-sync
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' relibc-sync
     command  unison -testserver /app_sync "socket://127.0.0.1:32787"
/Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/lib/docker-sync/sync_strategy/unison.rb:193:in `block in start_container': Failed to start unison container in time, try to increase max_attempt (currently 30) in your configuration. See https://github.com/EugenMayer/docker-sync/wiki/2.-Configuration for more informations (RuntimeError)

_Sync container logs_

user with uid 0 already exist and its root
/entrypoint.sh: ignoring /docker-entrypoint.d/*
/usr/lib/python2.7/site-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
Error: Format string '%(ENV_APP_VOLUME)s' for 'program:unison.directory' contains names ('ENV_APP_VOLUME') which cannot be expanded. Available names: ENV_GROUP, ENV_HOME, ENV_HOSTNAME, ENV_LANG, ENV_OWNER, ENV_OWNER_HOMEDIR, ENV_PATH, ENV_PWD, ENV_SHLVL, ENV_TZ, ENV_UNISON_DIR, ENV_UNISON_EXCLUDES, ENV_VOLUME, group_name, here, host_node_name, process_num, program_name in section 'program:unison' (file: '/etc/supervisor.conf.d/supervisor.daemon.conf')
For help, use /usr/bin/supervisord -h

Docker Driver

d4m (d4m or docker-machine+vbox/fusion)

Sync strategy

unison (default/native_osx/unison/rsync)

your docker-sync.yml

version: "2"

options:
  verbose: true
  max_attempt: 30
syncs:
  relibc-sync:
    src: '.'
    sync_strategy: 'unison'
    sync_excludes: ['target']

OS

OSX 10.13.3 (OSX Version/Linux)

[EugenMayer]

please ensure you

gem uninstall docker-sync
gem install docker-sync

then you run

docker-sync clean on your project

and then you remove your global configuration ~/.docker-sync-global.yml and then use start so the image is repulled

report back, if that does not fix it. please paste the outputs of each commands i gave above, thank you

[jeizsm]

Same. 0.5.2 version work perfectly

➜  relibc git:(master) gem uninstall docker-sync
                       gem install docker-sync
Remove executables:
    docker-sync, docker-sync-stack, docker-sync-daemon

in addition to the gem? [Yn]  y
Removing docker-sync
Removing docker-sync-stack
Removing docker-sync-daemon
Successfully uninstalled docker-sync-0.5.4
Fetching: docker-sync-0.5.4.gem (100%)
Successfully installed docker-sync-0.5.4
Parsing documentation for docker-sync-0.5.4
Installing ri documentation for docker-sync-0.5.4
Done installing documentation for docker-sync after 1 seconds
1 gem installed
➜  relibc git:(master) docker-sync clean
     success  Finished cleanup. Removed stopped, removed sync container and removed their volumes
➜  relibc git:(master) rm ~/.docker-sync-global.yml
➜  relibc git:(master) docker-sync start
          ok  Starting unison for sync relibc-sync
          ok  relibc-sync container not running
          ok  creating relibc-sync container
     command  docker run -p '127.0.0.1::5000' -v relibc-sync:/app_sync -e VOLUME=/app_sync -e TZ=${TZ-`readlink /etc/localtime | sed -e 's,/usr/share/zoneinfo/,,'`} -e UNISON_EXCLUDES="-ignore='Name target'"  --name relibc-sync -d eugenmayer/unison:0.3
          ok  starting initial sync of relibc-sync
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' relibc-sync
     command  unison -testserver /app_sync "socket://127.0.0.1:32792"
/Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/lib/docker-sync/sync_strategy/unison.rb:193:in `block in start_container': Failed to start unison container in time, try to increase max_attempt (currently 30) in your configuration. See https://github.com/EugenMayer/docker-sync/wiki/2.-Configuration for more informations (RuntimeError)
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/lib/docker-sync/sync_strategy/unison.rb:188:in `loop'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/lib/docker-sync/sync_strategy/unison.rb:188:in `start_container'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/lib/docker-sync/sync_process.rb:105:in `start_container'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/lib/docker-sync/sync_manager.rb:99:in `block in start_container'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/lib/docker-sync/sync_manager.rb:98:in `each'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/lib/docker-sync/sync_manager.rb:98:in `start_container'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/tasks/sync/sync.thor:163:in `daemonize'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/tasks/sync/sync.thor:47:in `start'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
    from /Users/jeiz/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/docker-sync-0.5.4/bin/docker-sync:14:in `<top (required)>'
    from /Users/jeiz/.rbenv/versions/2.3.3/bin/docker-sync:22:in `load'
    from /Users/jeiz/.rbenv/versions/2.3.3/bin/docker-sync:22:in `<main>'

Container logs

user with uid 0 already exist and its root
/entrypoint.sh: ignoring /docker-entrypoint.d/*
/usr/lib/python2.7/site-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
Error: Format string '%(ENV_APP_VOLUME)s' for 'program:unison.directory' contains names ('ENV_APP_VOLUME') which cannot be expanded. Available names: ENV_GROUP, ENV_HOME, ENV_HOSTNAME, ENV_LANG, ENV_OWNER, ENV_OWNER_HOMEDIR, ENV_PATH, ENV_PWD, ENV_SHLVL, ENV_TZ, ENV_UNISON_DIR, ENV_UNISON_EXCLUDES, ENV_VOLUME, group_name, here, host_node_name, process_num, program_name in section 'program:unison' (file: '/etc/supervisor.conf.d/supervisor.daemon.conf')
For help, use /usr/bin/supervisord -h

[CorentinDeBoisset]

Found the error, it is because docker-sync now uses the 0.3 unison container and not latest. The VOLUME environment variable was since replaced by APP_VOLUME, see this commit.

Opening a PR asap because I have the same issue.

[CorentinDeBoisset]

I would advise to either update or remove the latest tag on an outdated docker image, i had a hard time understanding what was going on.

[EugenMayer]

@coreoas good catch, thank you for investigating!

[EugenMayer]

@coreoas could you please verify this is a fix by installing docker-sync locally using ./deploy_locallly.sh and then clean / start? thank you. I will release .5 ASAP then

[CorentinDeBoisset]

Sure ! Giving you some feedback as soon as i'm done

[EugenMayer]

@coreoas removed :latest on hub, thanks for pointing out

[CorentinDeBoisset]

I am having a hard time starting the unison container... The entrypoint.sh does not match the latest here :/

This is the one in the container 0.3:

#!/usr/bin/env bash
set -e

if [ "$1" == 'supervisord' ]; then
    ################### ################### ###################
    ################### general core shared ###################
    ################### ################### ###################
    APP_VOLUME=${APP_VOLUME:-/app_sync}
    HOST_VOLUME=${HOST_VOLUME:-/host_sync}
    OWNER_UID=${OWNER_UID:-0}
    GROUP_ID=${GROUP_ID:-0}

    [ ! -d $APP_VOLUME ] && mkdir -p $APP_VOLUME

    # if the user did not set anything particular to use, we use root
    # since this means, no special user has been created on the target container
    # thus it is most probably root to run the daemon and thats a good default then
    if [ -z $OWNER_UID ];then
       OWNER_UID=0
    fi

    if [ ! -z $GROUP_ID ]; then

       # If gid doesn't exist on the system
       if ! cut -d: -f3 /etc/group | grep -q $GROUP_ID; then
           echo "no group has gid $GROUP_ID"
                 groupadd -g $GROUP_ID dockersync
       fi
    else
        GROUP_ID=0
    fi

    # if the user with the uid does not exist, create it, otherwise reuse it
    if ! cut -d: -f3 /etc/passwd | grep -q $OWNER_UID; then
        echo "no user has uid $OWNER_UID"

        # If user doesn't exist on the system
        useradd -u $OWNER_UID -g $GROUP_ID dockersync -m
    else
        if [ $OWNER_UID == 0 ]; then
            # in case it is root, we need a special treatment
            echo "user with uid $OWNER_UID already exist and its root"
        else
            # we actually rename the user to unison, since we do not care about
            # the username on the sync container, it will be matched to whatever the target container uses for this uid
            # on the target container anyway, no matter how our user is name here
            echo "user with uid $OWNER_UID already exist"
            existing_user_with_uid=$(awk -F: "/:$OWNER_UID:/{print \$1}" /etc/passwd)
            OWNER=`getent passwd "$OWNER_UID" | cut -d: -f1`
            GROUP=`getent group "$GROUP_ID" | cut -d: -f1`
            mkdir -p /home/$OWNER
            usermod -u $OWNER_UID -g $GROUP_ID $OWNER
            usermod --home /home/$OWNER $OWNER
            chown -R $OWNER /home/$OWNER
            chgrp -R $GROUP /home/$OWNER
         fi
    fi
    export OWNER_HOMEDIR=`getent passwd $OWNER_UID | cut -f6 -d:`
    # OWNER should actually be dockersync in all cases the user did not match a system user
    export OWNER=`getent passwd "$OWNER_UID" | cut -d: -f1`
    export GROUP=`getent group "$GROUP_ID" | cut -d: -f1`
# !! It crashes here because $VOLUME is not defined
    chown -R $OWNER $VOLUME
    chgrp -R $GROUP $VOLUME

    # see https://wiki.alpinelinux.org/wiki/Setting_the_timezone
    if [ -n ${TZ} ] && [ -f /usr/share/zoneinfo/${TZ} ]; then
        ln -sf /usr/share/zoneinfo/${TZ} /etc/localtime
        echo ${TZ} > /etc/timezone
    fi

    # Check if a script is available in /docker-entrypoint.d and source it
    for f in /docker-entrypoint.d/*; do
        case "$f" in
            *.sh)     echo "$0: running $f"; . "$f" ;;
            *)        echo "$0: ignoring $f" ;;
        esac
    done
    ################### ################### ###################
    ################### / general core shared/ ################
    ################### ################### ###################

    ################### ################### ###################
    ###################  now unison specific ###################
    ################### ################### ###################
    # Increase the maximum watches for inotify for very large repositories to be watched
    # Needs the privilegied docker option
    [ ! -z $MAX_INOTIFY_WATCHES ] && echo fs.inotify.max_user_watches=$MAX_INOTIFY_WATCHES | tee -a /etc/sysctl.conf && sysctl -p || true
    ################### ################### ###################
    ################### /now unison specific/ ###################
    ################### ################### ###################
    chown -R $OWNER_UID /unison
    chown $OWNER_UID /tmp/unison.log
fi

exec "$@"

There are some use of the now non-existant $VOLUME variable, and it crashes.

[EugenMayer]

you are looking at the wrong branch
https://github.com/EugenMayer/docker-image-unison/tree/develop

https://github.com/EugenMayer/docker-image-unison/blob/develop/entrypoint.sh

i will clean up that repo ASAP but for now, its a pile of... and a mess of branches and intensions :)

[CorentinDeBoisset]

Ohhh ok sorry ^^

Some pull request incoming then ;)

[CorentinDeBoisset]

There are some fixes needed to make unison work from the container.

First, unison v2.51 is required on the host, so I opened a PR on homebrew (this version wasn't available yet)

Now I need to also fix the way unison is started by supervisor. As of right now, it is set to listen from /host_sync (for a use with the native_osx strategy) and not with -socket 5000.

Need to find a way that does not crashes all the default configs... 🤔

[EugenMayer]

hmm, how did that go so mixed up, seems like this is a result of the overall mess in the unison repo

[EugenMayer]

There is an alternative way to install 2.51 using a graphical install now, but your PR is great to, added the graphical one to the docs.

For the image, please let me deal with that, i`ll jump to the rescue. I cause that, i will fix that mess, which means, i wil merge hostsync and the socket based unison into one single solution now

[CorentinDeBoisset]

Well, I saw your comment just now sorry, I already added a PR on your docker image:

https://github.com/EugenMayer/docker-image-unison/pull/15

It's a bit messy, it's basically a merge of develop and host_sync together into master, with some additional fixes to work in any configuration with docker-sync (see the 3 commits i made at the end).

This repo also needed an update, see #544.

I tested on my machine with the following setup:

version: "2"
options:
    verbose: true
    project_root: config_path

syncs:
    code_sync:
        sync_strategy: unison
        src: ./
        sync_excludes: [var/cache, var/logs, var/sessions]
        sync_userid: 1234

with both unison and native_osx strategies and it worked like a charm !

[EugenMayer]

thank you a lot @coreoas for you great work. #544 has been merged and a complete new unison image has been created, also by the work of @coreoas - it has been merged with the native_osx unison image to avoid such issues in the future, also the repo was reset and cleaned up and relocated.

the old repo is at https://github.com/EugenMayer/docker-image-unison-legacy .. the new one is at https://github.com/EugenMayer/docker-image-unison

I will most probably remove https://github.com/EugenMayer/docker-image-unison-legacy in the near future.

Since the homebrew PR has been merged now, unison 2.51.2 can now be installed / upgraded using homebrew

brew update
brew upgrade unison

This is now also part of the update check of 0.5.5 it will give this as notification during the update.
Hope this finally sorts things out and finally merged both images and efforts.

Great work @coreoas and thank you for the help, sorry at all the others, way much more testing should have gone into this before the release. Thats what happens when you quickly merge in pr because of an unison update, that forget about some people using it on OSX too, which means an update there - and also completely wrong PR bases.

Overall a mess, also poorly handled by me. Lets hope i can avoid this the next time.

[EugenMayer]

0.5.5 has been released

[michalkleiner]

Don't be that hard on yourself @EugenMayer, you're doing a great job and we all appreciate your time and effort put into this project. Thank you!

[michalkleiner]

I haven't found a way how to donate (on the website nor on GitHub) to support the development, can you point me in the right direction?

[EugenMayer]

@michalkleiner thanks for the heads up - mistakes are made to learn, so thats what i am looking at.

The best way to donate right now is by simply helping out, be it code, docs, or just helping in the issue queue. In any way, thank you very much for considering giving back, very appritated!