Distribution: Catalog is created with wrong user and group root:root inside Docker

[afiskon]

We use Docker in our Jepsen tests and observe a very weird behavior. Sometimes catalogs inside the container are created with wrong user and group root:root with no obvious reason. For instance:

go: gopkg.in/[email protected]: mkdir /home/gopher/go/pkg/mod/cache: permission denied
go: error loading module requirements
Makefile:60: recipe for target 'install-godep' failed
make: *** [install-godep] Error 1
gopher@c8e8aeac7f36:~/go/src/github.com/insolar/insolar$ ls -la /home/gopher/go/pkg/
total 20
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 16:01 .
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 15:38 ..
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 15:39 dep
drwxr-xr-x. 2 root   root   4096 Apr 19 16:01 mod
gopher@c8e8aeac7f36:~/go/src/github.com/insolar/insolar$ rm -r /home/gopher/go/pkg/mod
rm: remove write-protected directory '/home/gopher/go/pkg/mod'? y
gopher@c8e8aeac7f36:~/go/src/github.com/insolar/insolar$ mkdir /home/gopher/go/pkg/mod
gopher@c8e8aeac7f36:~/go/src/github.com/insolar/insolar$ ls -la /home/gopher/go/pkg/
total 20
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 16:02 .
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 15:38 ..
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 15:39 dep
drwxr-xr-x. 2 root   root   4096 Apr 19 16:02 mod

Steps to reproduce - follow the steps according to README.md in this repository https://github.com/insolar/insolar-jepsen

Environment:

$ docker version
Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:43:34 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:13:40 2019
  OS/Arch:          linux/amd64
  Experimental:     false

$ sudo kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

$ uname -a
Linux fake-host-name 4.4.178-1.el7.elrepo.x86_64 #1 SMP Wed Apr 3 05:46:30 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

[RomanSteinberg]

Hi,

there is a hypothesis that the problem occurred because of your Dockerfile line 27 and it is not a docker issue. You switched into root to get ssh access and after that all actions would take place with root privileges in the container.

Consider, another workflow. Run services as a root and then switch into gopher user. Another recommendations:

• don't use sudo docker commands, use gosu or su-exec,
• use entrypoints.

PS: #devzen calls for collaboration better then open source :smile:

[afiskon]

@RomanSteinberg

Thank you for the reply. This is an interesting hypothesis however I'm afraid it's most likely not right.

The problem appeared on the last line of https://github.com/insolar/insolar-jepsen/blob/master/base-image/Dockerfile:

RUN make install-deps pre-build

By like 27 you probably meant the last line of https://github.com/insolar/insolar-jepsen/blob/master/Dockerfile

CMD ["/usr/bin/sudo", "/usr/sbin/sshd", "-D"]

However this Dockerfile uses the first one as a base image. Thus this line couldn't cause the problem because it wasn't executed yet.

Also it worth noticing that the problem doesn't occur on Docker Desktop Community for MacOS 2.0.0.3 (31259), nor does it occur on Ubuntu 18.04 LTS with Docker 18.09.2 and Kubernetes 1.14. Which suggests that Dockerfiles are most likely OK.

We observed the issue only in the environment described in the first message.

[RomanSteinberg]

By line 27 you probably meant the last line of https://github.com/insolar/insolar-jepsen/blob/master/Dockerfile

Right. I forgot to specify the file.

[RomanSteinberg]

@afiskon
It looks like my hypothesis was wrong according to your last answer. But the example you provided is not minimal as soon as you have files which are not even executed (https://github.com/insolar/insolar-jepsen/blob/master/Dockerfile). So, I would like to suggest you to make a minimal example as much as it possible, because it will help people to reproduce the problem faster (obvious I think). Good luck!