Docker-sync: Rsync: Permission Issue "invalid uid"

[lucasjahn]

Hey there,

im running in a permission issue with the rsync mode. I have to use user name and user id to assign this files correctly to get synced to the fpm container.

The setup is as following:

compose.yml

version: '2'

services:
  mariadb:
    build:
      context: .
      dockerfile: build/env/mysql/Dockerfile
    ports:
      - 3306:3306
    environment:
       MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
       MYSQL_DATABASE: ${MYSQL_DATABASE}
       MYSQL_USER: ${MYSQL_USER}
       MYSQL_PASSWORD: ${MYSQL_PASSWORD}
    volumes:
      - "dev-database:/var/lib/mysql"


  php:
    build:
      context: .
      dockerfile: build/env/php-fpm/Dockerfile-dev
    env_file: .env
    volumes:
      - atf-rsync-sync:/var/www/html:rw
    links:
      - mariadb:mysql


  web:
    build:
      context: .
      dockerfile: build/env/nginx/Dockerfile-dev
    links:
      - php:fpm
    depends_on:
      - php
    volumes_from:
      - php
    environment:
      - PHP_IDE_CONFIG="serverName=localhost"
    ports:
      - 8070:80

volumes:
  dev-database:
    driver: local
  atf-rsync-sync:
    external: true

and the docker-sync.yml:

# rsync for faster development on osx
options:
  compose-file-path: 'base.yml'
  compose-dev-file-path: 'dev.yml'

syncs:
  atf-rsync-sync:
    notify_terminal: true
    sync_strategy: 'rsync'
    sync_user: 'www-data'
    sync_userid: '33'
    sync_group: 'www-data'
    sync_groupid: '33'
    src: './wordpress'
    dest: '/var/www/html'
    sync_host_ip: 'localhost'
    sync_host_port: 10872

if I start a clean docker-sync container it seems to work. If I then start my compose.yml and start saving files on the host. I get the exit code 5 "invalid uid"

In the docker-sync.yml I checked the permissions and the owner is xfs with the id 33. On my fpm container it is www-data with id 33. So somehow the docker-sync doesn't get the username correctly or cant update it to www-data and sync the files.

Has anyone a clue on this?

[EugenMayer]

Well that could actually be a bug. Could you ssh into the sync-container and check if /etc/passwd does actually include your www-data user - it should have been created on the sync-container uppon container start

[EugenMayer]

For reference: https://github.com/EugenMayer/docker-image-rsyncd/blob/master/run#L11

to ssh into the container docker --exec -it <containername> sh

[anthwinter]

I have resolved a similar issue with mapping permissions.

This is what I had originally (didn't work):

Dockerfile:

RUN groupadd docker && useradd -m -r -g docker docker
RUN chown -R docker /var/www/pt-webclient

docker-sync.yml:

syncs:
  webclient-rsync:
    src: '.'
    dest: '/var/www/pt-webclient'
    ...
    ...
    sync_user: 'docker'
    sync_group: 'docker'

I now have the uid/gid specificly set (this works)...

Dockerfile:

RUN groupadd -g 3000 docker && useradd -m -r -u 2000 -g docker docker
RUN chown -R docker /var/www/pt-webclient

docker-sync.yml:

syncs:
  webclient-rsync:
    src: '.'
    dest: '/var/www/pt-webclient'
    ...
    ...
    sync_user: 'docker'
    sync_userid: 2000
    sync_group: 'docker'
    sync_groupid: 3000

[spras]

I had a similar issue, can't use www-data user with it's usual ID 33, because it's already used by xfs user in the sync container.
Finally i understood how sync_user and sync_userid works, the www-data user is created in sync container, because it sat an unused userID in the sync container, AND in my app container.

But my problem seems now to be on the synced files. The Volume dir as the right user an userid, but the synced files belongs to nobody ...

/var/www/html # ls -la total 256 drwxr-xr-x 8 www-data www-data 4096 Nov 25 14:48 . drwxr-xr-x 3 root root 4096 Dec 8 10:02 .. -rw-r--r-- 1 nobody nogroup 10 Nov 25 14:48 .gitignore drwxr-xr-x 4 nobody nogroup 4096 Nov 25 12:37 app drwxr-xr-x 2 nobody nogroup 4096 Nov 25 14:21 bin -rw-r--r-- 1 nobody nogroup 3508 Nov 25 12:37 composer.json -rw-r--r-- 1 nobody nogroup 209949 Nov 25 14:21 composer.lock drwxr-xr-x 2 nobody nogroup 4096 Nov 25 12:37 src drwxr-xr-x 47 nobody nogroup 4096 Nov 25 14:21 vendor drwxr-xr-x 5 nobody nogroup 4096 Dec 7 17:32 web

in the sync container the rsync config file seems right :
/var/www/html # cat /etc/rsyncd.conf pid file = /var/run/rsyncd.pid uid = www-data gid = www-data use chroot = no log file = /dev/stdout reverse lookup = no munge symlinks = no [volume] hosts deny = * hosts allow = 192.168.0.0/16 172.16.0.0/12 read only = false path = /var/www/html comment = docker volume

I don't know what to do more ?

[EugenMayer]

the problem is that if uids are used, which are already taken in the unison container ( by system user ) things go bad.

xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin

I think that should be marked as a bug, not sure how we will address this so, it will need to have a cross referenced issue in https://github.com/EugenMayer/docker-image-unison-unox

[EugenMayer]

@spras that "ls" you showed, were is it from, the target container or the sync container? If it is the target container, you might just used the wrong uid which does not exist on the target container.

To ensure this is straight, for user/uid you use what you have on the target container or what you want there to be the owner, you do not care about the sync container - it should adjust and handle this for you

[spras]

@EugenMayer : Hi, this "ls" is from sync container.

Ok, I use www-data for user and 1000 for userid in both sync container, and target container.
Don't know if it's a bug, but it should be specified on documentation, that we might not use system UIDs like 33 (seen in /etc/passwd).

It seem i have a problem with rsync copying files with nobody as owner...

Thank's

[EugenMayer]

Are you using rsync or unison?

[spras]

rsync ... can't manage to start unison container for now

[EugenMayer]

So wait, ok rsync. Does it work with 1000/www-data for you? But it does not work with 33? which user is 33 in your target container? open a new issue for your unison issue, pretty sure we can fix that

[spras]

In docker-sync.yml,
with this config :
sync_user: 'www-data' sync_group: 'www-data' sync_userid: '33' sync_groupid: '33'
doesn't work at all, it "normal" because the user xfs already use this userid

with this config :
sync_user: 'www-data' sync_group: 'www-data'
the user www-data is created, with defaut uid (1000), it's OK , but the rsync create files with owner nobody

with this config :
sync_user: 'www-data' sync_group: 'www-data' sync_userid: '1000' sync_groupid: '1000'
the user www-data is created, with uid (1000), it's OK , but the rsync create files with owner nobody

My problem is realy on rsynced files owner.

Don't understand why because, the rsync config file seem right :

````
/var/www/html # cat /etc/rsyncd.conf
pid file = /var/run/rsyncd.pid
uid = www-data
gid = www-data
use chroot = no
log file = /dev/stdout
reverse lookup = no
munge symlinks = no
[volume]
hosts deny = *
hosts allow = 192.168.0.0/16 172.16.0.0/12
read only = false
path = /var/www/html
comment = docker volume
`````

[EugenMayer]

Ah, that now makes sense!

https://github.com/EugenMayer/docker-image-rsyncd/blob/master/run#L16

This means, that https://github.com/EugenMayer/docker-image-rsyncd/blob/master/run#L14 can fail, but still the chown is done.

This is a bug, please open a bug request in the rsyncd issue and we work this out.

Thank you for your patience!

[EugenMayer]

fixed

[teebszet]

Hi, I have a situation where I'm syncing with the option sync_userid: 48, but getting:

-rw-r--r--  1 apache  65534 434798 Feb  6 08:08 CHANGELOG.md
-rw-r--r--  1 apache  65534   3671 Feb  6 08:08 CONTRIBUTING.md
-rw-r--r--  1 apache  65534    647 Feb  6 08:08 COPYING.txt

where I want the user:group to be apache:apache

If I set the sync_groupid: 48 option as suggested by examples above, then I get a runtime error:

/usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_strategy/rsync.rb:129:in `get_group_mapping': for now, rsync does no longer support groupid, but for nearly all cases sync_userid should be enough (RuntimeError)
    from /usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_strategy/rsync.rb:98:in `start_container'
    from /usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_process.rb:105:in `start_container'
    from /usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_manager.rb:99:in `block in start_container'
    from /usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_manager.rb:98:in `each'
    from /usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_manager.rb:98:in `start_container'
    from /usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/tasks/sync/sync.thor:163:in `daemonize'
    from /usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/tasks/sync/sync.thor:47:in `start'
    from /usr/local/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
    from /usr/local/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
    from /usr/local/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
    from /usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/bin/docker-sync:14:in `<top (required)>'
    from /usr/local/bin/docker-sync:22:in `load'
    from /usr/local/bin/docker-sync:22:in `<main>'

what's the correct way to specify user:group for the rsync target?

[EugenMayer]

/usr/local/lib/ruby/gems/2.4.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_strategy/rsync.rb:129:in `get_group_mapping': for now, rsync does no longer support groupid, but for nearly all cases sync_userid should be enough (RuntimeError)

That tells you groupid is not supported, so its not supported :)