Docker.github.io: docker user namespace remap not working?

[Bharathkumarraju]

I have tried to remap the user namespaces in docker but getting the process_linux.go:297 error.

docker: Error response from daemon: OCI runtime create failed: container_linux.go:337: starting container process caused "process_linux.go:297: getting the final child's pid from pipe caused \"EOF\"": unknown.

[root@test docker]# docker run -itd openjdk sh -c 'sleep 500'
Unable to find image 'openjdk:latest' locally
latest: Pulling from library/openjdk
cd8eada9c7bb: Pull complete
c2677faec825: Pull complete
fcce419a96b1: Pull complete
045b51e26e75: Pull complete
88e50f3a5916: Pull complete
9db1045008ba: Pull complete
5ba72089e00c: Pull complete
810bdb5dd91f: Pull complete
62b563475556: Pull complete
Digest: sha256:e01aa552356f6f78a4bf2dd3576874c3e7b58c64cce0cc5bf1d538911d2dc86e
Status: Downloaded newer image for openjdk:latest
86df703cd7cbdff86d3f9ec94eedaefeb335939489b12971c3a99345b7356c9f
docker: Error response from daemon: OCI runtime create failed: container_linux.go:337: starting container process caused "process_linux.go:297: getting the final child's pid from pipe caused \"EOF\"": unknown.
[root@test docker]#

[root@test docker]# id bharath
uid=1000(bharath) gid=1000(bharath) groups=1000(bharath),10(wheel),995(docker)
[root@test docker]# cat /etc/subuid
bharath:165536:65536
[root@test docker]# cat /etc/subgid
bharath:165536:65536
[root@test docker]# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd  --userns-remap=bharath -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this option.
TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target
[root@test docker]#

docker service status:

[root@test docker]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-01-02 15:46:54 CET; 6s ago
     Docs: https://docs.docker.com
 Main PID: 17375 (dockerd)
    Tasks: 9
   Memory: 29.8M
   CGroup: /system.slice/docker.service
           └─17375 /usr/bin/dockerd --userns-remap=bharath -H fd://

Jan 02 15:46:54 test.docker dockerd[17375]: time="2019-01-02T15:46:54.332117934+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420045250, READY" module=grpc
Jan 02 15:46:54 test.docker dockerd[17375]: time="2019-01-02T15:46:54.348374120+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jan 02 15:46:54 test.docker dockerd[17375]: time="2019-01-02T15:46:54.349333447+01:00" level=info msg="Loading containers: start."
Jan 02 15:46:54 test.docker dockerd[17375]: time="2019-01-02T15:46:54.610045331+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 02 15:46:54 test.docker dockerd[17375]: time="2019-01-02T15:46:54.779897651+01:00" level=info msg="Loading containers: done."
Jan 02 15:46:54 test.docker dockerd[17375]: time="2019-01-02T15:46:54.803369030+01:00" level=info msg="Docker daemon" commit=bca0068 graphdriver(s)=overlay2 version=18.09.1-rc1
Jan 02 15:46:54 test.docker dockerd[17375]: time="2019-01-02T15:46:54.803575980+01:00" level=info msg="Daemon has completed initialization"
Jan 02 15:46:54 test.docker dockerd[17375]: time="2019-01-02T15:46:54.809345018+01:00" level=warning msg="Could not register builder git source: failed to find git binary: exec: \"git\": executable file not found in $PATH"
Jan 02 15:46:54 test.docker dockerd[17375]: time="2019-01-02T15:46:54.818827761+01:00" level=info msg="API listen on /var/run/docker.sock"
Jan 02 15:46:54 test.docker systemd[1]: Started Docker Application Container Engine.
[root@test docker]#

my /var/lib/docker showed correctly as well.

[root@test docker]# cd /var/lib/docker
[root@test docker]# ls -rtlh
total 0
drwx------.  2 root   root     6 Jan  2 15:42 runtimes
drwx------.  4 root   root    32 Jan  2 15:42 plugins
drwx------.  3 root   root    22 Jan  2 15:42 image
drwx------.  2 root   root    25 Jan  2 15:42 volumes
drwx------.  2 root   root     6 Jan  2 15:42 trust
drwxr-x---.  3 root   root    19 Jan  2 15:42 network
drwx------.  2 root   root     6 Jan  2 15:42 swarm
drwx------.  2 root   root    24 Jan  2 15:42 builder
drwx------.  4 root   root    92 Jan  2 15:42 buildkit
drwx------.  2 root   root     6 Jan  2 15:42 tmp
drwx------.  6 root   root   261 Jan  2 15:42 overlay2
drwx------.  3 root   root    78 Jan  2 15:42 containers
drwx------. 14 165536 165536 182 Jan  2 15:46 165536.165536

[root@test docker]# cd 165536.165536/
[root@test 165536.165536]# ls -rtlh
total 4.0K
drwx------.  2 root   root      6 Jan  2 15:46 runtimes
drwx------.  4 root   root     32 Jan  2 15:46 plugins
drwx------.  3 root   root     22 Jan  2 15:46 image
drwx------.  2 165536 165536   25 Jan  2 15:46 volumes
drwx------.  2 root   root      6 Jan  2 15:46 trust
drwxr-x---.  3 root   root     19 Jan  2 15:46 network
drwx------.  2 root   root      6 Jan  2 15:46 swarm
drwx------.  2 root   root     24 Jan  2 15:46 builder
drwx------.  4 root   root     92 Jan  2 15:46 buildkit
drwx------.  2 165536 165536    6 Jan  2 15:48 tmp
drwx------. 14 165536 165536 4.0K Jan  2 15:48 overlay2
drwx------.  3 165536 165536   78 Jan  2 15:48 containers
[root@test 165536.165536]#

when I try to run the docker run it failed as below.

[root@test docker]# docker run -itd openjdk sh -c 'sleep 500'
Unable to find image 'openjdk:latest' locally
latest: Pulling from library/openjdk
cd8eada9c7bb: Pull complete
c2677faec825: Pull complete
fcce419a96b1: Pull complete
045b51e26e75: Pull complete
88e50f3a5916: Pull complete
9db1045008ba: Pull complete
5ba72089e00c: Pull complete
810bdb5dd91f: Pull complete
62b563475556: Pull complete
Digest: sha256:e01aa552356f6f78a4bf2dd3576874c3e7b58c64cce0cc5bf1d538911d2dc86e
Status: Downloaded newer image for openjdk:latest
86df703cd7cbdff86d3f9ec94eedaefeb335939489b12971c3a99345b7356c9f
docker: Error response from daemon: OCI runtime create failed: container_linux.go:337: starting container process caused "process_linux.go:297: getting the final child's pid from pipe caused \"EOF\"": unknown.
[root@test docker]#

[Bharathkumarraju]

ahh it got fixed by running the below command.

sysctl -w user.max_user_namespaces=15000

[wstian]

grep docker container on the node, to which the pod is assigned

docker ps -a |grep [pod-name]

a related pause container is in Created status， got the container id. then grep the
/var/log/meessage ， you will find the root cause, why pause container failed to create.

in our case, the error is page allocation failure, which it's due to low system free memory. To fix this,
according to this article https://discuss.aerospike.com/t/page-allocation-failure/2856

# free the caches
sync ; echo 3|tee /proc/sys/vm/drop_caches
# rm the container, the kubelet will restart the pod
docker rm <container-id>

to prevent, increase the free memory

sysctl -w vm.min_free_kbytes=5242880

[VictorLee0321]

systemctl restart docker.service works for me.
look at:
https://github.com/microsoft/vscode-docker/issues/1963#issuecomment-627254890