Runc: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""

Created on 23 Feb 2017  ·  38Comments  ·  Source: opencontainers/runc

Hi

we are seeing the below issue when trying to build an image using Dockerfile.

nsenter: failed to unshare namespaces: Invalid argument
container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""
oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""

can someone please address this?

picture reppala

Most helpful comment

Last version I can get to work on ubuntu 14.04 from https://download.docker.com/linux/ubuntu is:
apt-get -y install --force-yes docker-ce=18.06.1~ce~3-0~ubuntu

picture StevenACoffman on 1 Apr 2019
👍12 🎉4

All 38 comments

@rajscotia What version of runC is this with (what is the output of runc --version or docker-runc --version)?

cyphar picture cyphar on 24 Feb 2017

Hi Cyphar, thanks for your response. Here is the runc version.

[root@lvappi01089 ~]# runc --version
runc version 1.0.0-rc2
spec: 1.0.0-rc2-dev
[root@lvappi01089 ~]# docker-runc --version
runc version 1.0.0-rc2
commit: 9df8b306d01f59d3a8029be411de015b7304dd8f
spec: 1.0.0-rc2-dev

reppala picture reppala on 24 Feb 2017

[root@lvappi01089 ~]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.13.1
Storage Driver: devicemapper
Pool Name: vg_app-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 21.47 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 816.8 MB
Data Space Total: 1.071 TB
Data Space Available: 1.07 TB
Metadata Space Used: 1.442 MB
Metadata Space Total: 11.27 GB
Metadata Space Available: 11.27 GB
Thin Pool Minimum Free Space: 107.1 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
userns
Kernel Version: 3.10.0-514.6.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51 GiB
Docker Root Dir: /var/lib/docker/100000.100000
Debug Mode (client): false
Debug Mode (server): false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

[root@lvappi01089 ~]# docker version
Client:
Version: 1.13.1
API version: 1.26
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:38:28 2017
OS/Arch: linux/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:38:28 2017
OS/Arch: linux/amd64
Experimental: false
[root@lvappi01089 ~]#

reppala picture reppala on 24 Feb 2017

Looks like this is hitting our old friend

/*
 * Unshare all of the namespaces. Now, it should be noted that this
 * ordering might break in the future (especially with rootless
 * containers). But for now, it's not possible to split this into
 * CLONE_NEWUSER + [the rest] because of some RHEL SELinux issues.
 *
 * Note that we don't merge this with clone() because there were
 * some old kernel versions where clone(CLONE_PARENT | CLONE_NEWPID)
 * was broken, so we'll just do it the long way anyway.
 */
if (unshare(config.cloneflags) < 0)
    bail("failed to unshare namespaces");

@mrunalp And ideas on why this might be breaking on RHEL 7.3? [Hint: The answer is probably SELinux.]

cyphar picture cyphar on 25 Feb 2017

I don't see this on my RHEL 7 system. I suspect that an older kernel may be in play here.

On Feb 24, 2017, at 4:01 PM, Aleksa Sarai notifications@github.com wrote:

Looks like this is hitting our old friend

      if (unshare(config.cloneflags) < 0)
          bail("failed to unshare namespaces");

@mrunalp And ideas on why this might be breaking on RHEL 7.3?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

mrunalp picture mrunalp on 25 Feb 2017

Am seeing the identical error message on Centos 7.3 with SELinux disabled.

cat /etc/*-release

CentOS Linux release 7.3.1611 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOS Linux release 7.3.1611 (Core)
CentOS Linux release 7.3.1611 (Core)

sestatus

SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28

docker info

Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 3
Server Version: 17.03.1-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
userns
Kernel Version: 3.10.0-514.10.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.4 GiB
Name: bear.kupnet
ID: 62UK:EA55:CJQB:B36Q:NKHI:QZW7:QRTL:BBUX:6CQ5:W6JC:AUVA:QQYT
Docker Root Dir: /var/lib/docker/100000.100000
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

sudo cat /etc/docker/daemon.json

{
"graph": "/var/lib/docker",
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "2"
},
"debug": false,
"userns-remap": "django:django",
"iptables": false
}

sudo cat /etc/passwd | grep django

django: x:5000:5000::/home/django:/sbin/nologin

sudo cat /etc/group | grep django

django: x:5000:

sudo cat /etc/subuid

django:100000:65536

sudo cat /etc/subgid

django:100000:65536

After sudo systemctl restart docker.service
sudo systemctl status docker.service

● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2017-04-13 09:30:06 CDT; 41s ago
Docs: https://docs.docker.com
Main PID: 23252 (dockerd)
Memory: 12.8M
CGroup: /system.slice/docker.service
├─23252 /usr/bin/dockerd
└─23259 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/...
\
Apr 13 09:30:05 bear.kupnet dockerd[23252]: time="2017-04-13T09:30:05.708915124-05:00" level=warning msg="overlay: the backing xfs filesystem is formatted without d_type support, whi...
Apr 13 09:30:05 bear.kupnet dockerd[23252]: time="2017-04-13T09:30:05.708987071-05:00" level=info msg="[graphdriver] using prior storage driver: overlay"
Apr 13 09:30:06 bear.kupnet dockerd[23252]: time="2017-04-13T09:30:06.344136976-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Apr 13 09:30:06 bear.kupnet dockerd[23252]: time="2017-04-13T09:30:06.345537179-05:00" level=info msg="Loading containers: start."
Apr 13 09:30:06 bear.kupnet dockerd[23252]: time="2017-04-13T09:30:06.751238256-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/... IP address"
Apr 13 09:30:06 bear.kupnet dockerd[23252]: time="2017-04-13T09:30:06.950213189-05:00" level=info msg="Loading containers: done."
Apr 13 09:30:06 bear.kupnet dockerd[23252]: time="2017-04-13T09:30:06.963166530-05:00" level=info msg="Daemon has completed initialization"
Apr 13 09:30:06 bear.kupnet dockerd[23252]: time="2017-04-13T09:30:06.963214644-05:00" level=info msg="Docker daemon" commit=c6d412e graphdriver=overlay version=17.03.1-ce
Apr 13 09:30:06 bear.kupnet dockerd[23252]: time="2017-04-13T09:30:06.978585354-05:00" level=info msg="API listen on /var/run/docker.sock"
Apr 13 09:30:06 bear.kupnet systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.

Then using this Dockerfile

FROM ruby:2.3.2-slim
\
ENV RAILS_ROOT=/var/www/passkeeper2
\
RUN apt-get update -qq && \
apt-get install -y build-essential libpq-dev postgresql-client git && \
apt-get clean autoclean && \
apt-get autoremove -y && \
mkdir -p $RAILS_ROOT/tmp/pids
\
WORKDIR $RAILS_ROOT
\
COPY Gemfile Gemfile
\
COPY Gemfile.lock Gemfile.lock
\
RUN gem install bundler && bundle install
\
COPY . .
\
CMD [ "config/containers/app_cmd.sh" ]

docker build . yields this:

Sending build context to Docker daemon 15.67 MB
Step 1/9 : FROM ruby:2.3.2-slim
---> 7e109fd4329e
Step 2/9 : ENV RAILS_ROOT /var/www/passkeeper2
---> Using cache
---> 94e4dbed2329
Step 3/9 : RUN apt-get update -qq && apt-get install -y build-essential libpq-dev postgresql-client git && apt-get clean autoclean && apt-get autoremove -y && mkdir -p $RAILS_ROOT/tmp/pids
---> Running in c67a4fd5599a
nsenter: failed to unshare namespaces: Invalid argument
container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""
oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""

FWIW: Rebooting the host has no effect on this.

dpneumo picture dpneumo on 13 Apr 2017

Re-reading the above I noticed that I am getting a warning from dockerd in the docker system status report.

dockerd[5321]: time="2017-04-13T10:11:51.054545432-05:00" level=warning msg="overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type support will no longer be supported in Docker 17.12."

Since I am on docker server version 17.03.1-ce one might think this is not yet a problem. Still the warning about incorrect behaviour raises the possibility that running on a "backing xfs filesystem [that] is formatted without d_type support" is involved in the namespace error.

My host is a development box that is heavily used for many projects. Am not willing to reformat the file system without careful preparation. If anyone else has any thoughts on the contribution of the file system "without d_type support" to the namespace problem would be interested to hear them.

From mkfs.xfs manpage:

     mkfs.xfs ... [ -n naming_options ] ...

        -n naming_options:
             These options specify the version and size parameters for the
             naming (directory) area of the filesystem. The valid
             naming_options are:
                 ...
                 ftype=value
                      This feature allows the inode type to be stored in
                      the directory structure so that the readdir(3) and
                      getdents(2) do not need to look up the inode to
                      determine the inode type.

                      The value is either 0 or 1, with 1 signifying that
                      filetype information will be stored in the
                      directory structure.  The default value is 1.

                      When CRCs are enabled (the default), the ftype
                      functionality is always enabled, and cannot be
                      turned off.
dpneumo picture dpneumo on 13 Apr 2017

Have you ever try to use another graphdriver? Just to make sure whether this namespace error is related to backing xfs filesystem that is formatted without d_type support.

Besides, using backing filesystem without d_type support for overlay/overlay2 will also cause other problems (e.g #27358). So I think it's better to reformat the file system if it's possible.

yummypeng picture yummypeng on 14 Apr 2017

@yummypeng Thanks for the suggestion. To test this I changed the storage driver to vfs:

/etc/docker/daemon.json

{
"graph": "/var/lib/docker",
"storage-driver": "vfs",
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "2"
},
"debug": false,
"iptables": false
}

docker build .

completed successfully.

I think that makes it more likely that the namespace error is related to the backing xfs filesystem that is formatted without d_type support.

Adding

"userns-remap": "django:django",

to daemon.json and restarting the docker service caused docker build . to fail:

Step 3/9 : RUN apt-get update -qq ...
---> Running in 5e591351381a
nsenter: failed to unshare namespaces: Invalid argument
container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""
oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""

So it seems the storage driver is not be the problem here. The backing filesystem, xfs, with its faulty setup remains a likely suspect.

I do plan to reformat the file system. Must take steps to assure that I do not shoot myself in the foot when I do so.

Will do a quick trial of creating an xfs partition with appropriate support. If that avoids the namespace error I probably should submit a PR to add a warning in the Docker install docs for CentOS based hosts. RHEL7 and CentOS7 appear to default to xfs. I did the original install by allowing the default partitioning. It seems the default partitioning does NOT provide "d_type" support. Would expect others to fall into this trap as a consequence.

Wonder if I would be better off with ext4? Gotchas with that?

dpneumo picture dpneumo on 14 Apr 2017

BTW. For anyone else reading this, vfs is not a union file system and is not appropriate for production use.

Storage Drivers in Docker: A Deep Dive
Docker User Guide - storage driver selection

dpneumo picture dpneumo on 14 Apr 2017
👍1

@dpneumo

nsenter: failed to unshare namespaces: Invalid argument

Are you sure your kernel supports CLONE_NEWUSER? I know that RHEL/CentOS has some weird sysctls they added to allow administrators to disable certain features of user namespaces (including disabling them entirely even though CLONE_NEWUSER is compiled into the kernel).

Can you see if unshare --user works? What about zgrep USER_NS /proc/config.gz?

cyphar picture cyphar on 15 Apr 2017

@cyphar
Sorry for the slow response.

I think you must be correct. Am on CentOS 7.3.1611 with Kernel Version: 3.10.0-514.10.2.el7.x86_64 direct from the CentOS mirrors. Presumably either this kernel does not support CLONE_NEWUSER or some/all features have been disabled. I suppose I will have to shift to a different linux distro.

Before I saw your response I reformatted the partition:

xfs_info /home
meta-data=/dev/mapper/centos_bear-home isize=512 agcount=4, agsize=61046784 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=244187136, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=119232, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

But with:

sudo cat /etc/docker/daemon.json
{
"graph": "/var/lib/docker",
"storage-driver": "overlay",
"debug": false,
"userns-remap": "django:django",
"iptables": false
}

I get the same error:

nsenter: failed to unshare namespaces: Invalid argument
container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""
oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""

Per your question I tried as a normal user and as root:

unshare --map-root-user --user sh -c whoami
unshare: unshare failed: Invalid argument

and

unshare --user sh
unshare: unshare failed: Invalid argument

This was based on examples from the unshare man page. Variations in the unshare arguments were tried with the same result.

On your second question I assume you meant "zgrep $USER_NS /proc/config.gz". In the original form and running as root that returns:

gzip: /proc/config.gz: No such file or directory

or with $USER_NS it does not return at all.

ls -last /proc | grep config

does not find anything.

dpneumo picture dpneumo on 16 Apr 2017

@cyphar

I know that RHEL/CentOS has some weird sysctls they added to allow administrators to disable certain features of user namespaces

If CLONE_NEWUSER is compiled into my kernel is there a mechanism for enabling its features? Do you know how to discover available features of user namespaces? Enabled/disabled features?

dpneumo picture dpneumo on 16 Apr 2017

You need to enable it in the kernel command line in RHEL 7/Centos:

grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
mrunalp picture mrunalp on 16 Apr 2017
👎3 👍1

As @mrunalp said, you need to set some arguments on the kernel cmdline.

unshare --user sh
unshare: unshare failed: Invalid argument

Yeah, your kernel isn't allowing CLONE_NEWUSER.

On your second question I assume you meant "zgrep $USER_NS /proc/config.gz".

No I didn't. zgrep USER_NS /proc/config.gz will search for the string USER_NS in /proc/config.gz -- though I didn't know that RHEL doesn't enable this pseudo-file way of figuring out the kenrel configuration

cyphar picture cyphar on 17 Apr 2017

@mrunalp
Yea!! That works.

I had just found this: How to enable "user" namespace in RHEL7 and CentOS7? when your comment came up.

@cyphar
Your help is greatly appreciated..

dpneumo picture dpneumo on 17 Apr 2017

@mrunalp @cyphar
Well I may have spoken too quickly. "docker build . " nol longer fails. However, if I use docker-compose to bring up the entire project I see this error:

ERROR: for app Cannot start service app: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\"/home/loco/Projects/passkeeper2\\" to rootfs \\"/var/lib/docker/100000.100000/overlay/ce472ee810d294f0e96237de20e74331ea77b014c2248e935f9f6d4c8569e72f/merged\\" at \\"/var/www/passkeeper2\\" caused \\"stat /home/loco/Projects/passkeeper2: permission denied\\"\""

This occurs as a normal user or as root. I will explore the permission issues. but wonder if either of you have any thoughts.

dpneumo picture dpneumo on 17 Apr 2017

Since everything from /var/lib/docker/100000.100000 on is owned by 100000 and has permissions 700 I wonder if everything in the project folder should be owned by django. Recall:

sudo cat /etc/passwd | grep django

django: x:5000:5000::/home/django:/sbin/nologin

sudo cat /etc/group | grep django

django: x:5000:

sudo cat /etc/subuid

django:100000:65536

sudo cat /etc/subgid

django:100000:65536

ls of the subject dirs:

ls -last /home/loco/Projects | grep passkeeper2
4 drwxrwxr-x. 13 loco loco 4096 Apr 9 22:47 passkeeper2

ls -last /var/www
total 4
4 drwxr-xr-x. 22 root root 4096 Apr 16 15:27 ..
0 drwxr-xr-x. 3 root root 17 Jun 21 2016 .

ls -last /var/lib/docker
total 28
4 drwxr-xr-x. 63 root root 4096 Apr 16 15:27 ..
0 drwx------. 2 root root 6 Apr 16 13:06 tmp
8 drwx------. 46 root root 4096 Apr 16 13:06 overlay
4 drwx------. 11 root root 4096 Apr 16 13:06 containers
0 drwx------. 5 root root 49 Apr 14 16:11 image
4 drwx--x--x. 14 root root 4096 Apr 14 16:11 .
4 drwx------. 12 100000 100000 4096 Apr 14 15:38 100000.100000
0 drwx------. 3 root root 16 Apr 14 14:16 vfs
4 drwx------. 6 root root 4096 Apr 10 00:02 volumes
0 drwx------. 4 root root 30 Jan 24 12:06 plugins
0 drwx------. 2 root root 6 Sep 2 2016 swarm
0 drwx------. 5 root root 50 Mar 16 2016 devicemapper
0 drwxr-x---. 3 root root 18 Mar 16 2016 network

>

ls -last /var/lib/docker/100000.100000
total 24
4 drwx------. 11 100000 100000 4096 Apr 16 17:17 containers
12 drwx------. 55 100000 100000 8192 Apr 16 17:17 overlay
0 drwx------. 2 100000 100000 6 Apr 16 17:17 tmp
4 drwx--x--x. 14 root root 4096 Apr 14 16:11 ..
0 drwx------. 3 100000 100000 16 Apr 14 15:39 vfs
4 drwx------. 12 100000 100000 4096 Apr 14 15:38 .
0 drwx------. 4 root root 30 Apr 14 15:38 image
0 drwx------. 4 100000 100000 85 Apr 13 00:45 volumes
0 drwx------. 2 root root 6 Apr 13 00:22 swarm
0 drwxr-x---. 3 root root 18 Apr 13 00:22 network
0 drwx------. 2 root root 6 Apr 13 00:22 trust
0 drwx------. 4 root root 30 Apr 13 00:22 plugins

ls -last /var/lib/docker/100000.100000/overlay | grep "Apr 16 17:17"
0 drwx------. 5 100000 100000 57 Apr 16 17:17 ce472ee...69e72f
12 drwx------. 55 100000 100000 8192 Apr 16 17:17 .
0 drwx------. 5 100000 100000 57 Apr 16 17:17 ce472ee...69e72f-init
0 drwx------. 5 100000 100000 57 Apr 16 17:17 268021f...10d740
0 drwx------. 5 100000 100000 57 Apr 16 17:17 268021f...10d740-init

ls -last /var/lib/docker/100000.100000/overlay/ce472ee...69e72f
total 16
0 drwx------. 3 100000 100000 17 Apr 16 17:17 work
0 drwx------. 5 100000 100000 57 Apr 16 17:17 .
0 drwx------. 2 100000 100000 6 Apr 16 17:17 merged
0 drwxr-xr-x. 4 100000 100000 43 Apr 16 17:17 upper
12 drwx------. 55 100000 100000 8192 Apr 16 17:17 ..
4 -rw-r--r--. 1 root root 64 Apr 16 17:17 lower-id

ls -last /var/lib/docker/100000.100000/overlay/ce472ee...69e72f/merged
total 0
0 drwx------. 2 100000 100000 6 Apr 16 17:17 .
0 drwx------. 5 100000 100000 57 Apr 16 17:17 ..

dpneumo picture dpneumo on 17 Apr 2017

Nope. Just need "world execute permissions in all the directories below the one you are mounting" #28859.

For my set up that meant chmod 755 /home /home/loco /home/loco/Projects
Then for good measure I did find passkeeper2 -type d -print0 | xargs -0 chmod 775

That allowed docker build . to succeed. There were some remaining issues but these were project specific.

To summarize for myself and others who might land here, I created this gist:
Setup User Namespaces on RHEL/Centos 7.3

dpneumo picture dpneumo on 17 Apr 2017

Hi
I run into the same behavior on a fresh centos 7.3 install whereas it works well on another centos 7.3 one.
The only difference was the kernel version.

According this trello, https://trello.com/c/Hw3Ss3EK/382-sysctl-maxusernamespaces-handled-for-docker-in-rhel-73, from kernel 3.10.0-644, you need to change the sysctl parameter user.max_user_namespace to a non zero value, which is the default.

Modifying the sysctl parameter did the trick for me

aolwas picture aolwas on 15 Dec 2017
👍1

I really want to use concourse ci for my team,but it is really difficult for me to solve the same problem.SOS,This is not very easy to get started with

gezhiwei8899 picture gezhiwei8899 on 18 Dec 2017

RUN echo 'deb http://mirrors.aliyun.com/debian/ stretch main non-free contrib \n \
deb-src http://mirrors.aliyun.com/debian/ stretch main non-free contrib \n \
deb http://mirrors.aliyun.com/debian-security stretch/updates main \n \
deb-src http://mirrors.aliyun.com/debian-security stretch/updates main \n \
deb http://mirrors.aliyun.com/debian/ stretch-updates main non-free contrib \n \
deb-src http://mirrors.aliyun.com/debian/ stretch-updates main non-free contrib \n \
deb http://mirrors.aliyun.com/debian/ stretch-backports main non-free contrib \n \
deb-src http://mirrors.aliyun.com/debian/ stretch-backports main non-free contrib' \

/etc/apt/sources.list \
&& apt update && apt install -y procps ca-certificates \
&& mkdir -p /data/resources/csv

cuishuang picture cuishuang on 3 Dec 2018
👍1

i have the same question ubuntu 18
The Prompt message is
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\"cgroup\\" to rootfs \\"/var/lib/docker/aufs/mnt/1c172f44436cd02f737dff1f183b127f521a1754da1e14201dd59de38b27eeaf\\" at \\"/sys/fs/cgroup\\" caused \\"no subsystem for mount\\"\".

newShiJ picture newShiJ on 1 Feb 2019

Same issue on Ubuntu 14.04

wandermyz picture wandermyz on 13 Feb 2019

Same issue on Ubuntu 14.04 too

$ docker system prune -a -f
Deleted Containers:
[...]
$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete 
Digest: sha256:2557e3c07ed1e38f26e389462d03ed943586f744621577a99efb77324b0fe535
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:297: copying bootstrap data to pipe caused \"write init-p: broken pipe\"": unknown.
ERRO[0002] error waiting for container: context canceled
piroux picture piroux on 21 Feb 2019
👍1

I had the same issue on Ubuntu 14.04 yesterday. Upgraded to 16.04 with sudo do-release-upgrade and started working. (the upgrade took about 2 hours and required input decisions constantly from 1 to 10min average). I'm not sure why, but I think is about 14.04 losing docker support.

bonafernando picture bonafernando on 13 Mar 2019

Last version I can get to work on ubuntu 14.04 from https://download.docker.com/linux/ubuntu is:
apt-get -y install --force-yes docker-ce=18.06.1~ce~3-0~ubuntu

StevenACoffman picture StevenACoffman on 1 Apr 2019
👍12 🎉4

Last version I can get to work on ubuntu 14.04 from https://download.docker.com/linux/ubuntu is:
apt-get -y install --force-yes docker-ce=18.06.1~ce~3-0~ubuntu

docker-ce=18.06.1~ce~3-0~ubuntu worked but 18.06.3~ce~3-0~ubuntu still have this problem

whlsxl picture whlsxl on 3 Apr 2019
👍4

docker-ce=18.06.1~ce~3-0~ubuntu worked

Harsha-Bidgely picture Harsha-Bidgely on 8 Apr 2019

On cent os 7.2.1511, produced after I upgrade docker from:
Docker version 1.13.1, build 6e3bb8e/1.13.1
to
Docker version 1.13.1, build b2f74b2/1.13.1

lufy90 picture lufy90 on 23 Apr 2019

Last version I can get to work on ubuntu 14.04 from https://download.docker.com/linux/ubuntu is:
apt-get -y install --force-yes docker-ce=18.06.1~ce~3-0~ubuntu

I also have the same issue on Ubuntu 14.04 when installing latest docker-ce version.

After downgrading the specific docker-ce version, I don't get this issue and it looks worked successfully now.

peter279k picture peter279k on 21 May 2019
👍1

met this issue
"docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:245: running exec setns process for init caused \\\"exit status 29

in 

Containers: 47
 Running: 31
 Paused: 0
 Stopped: 16
Images: 739
Server Version: 17.03.2-ce
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.14.4.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.51 GiB
Name: synthetic-fvt3-node31.fyre.ibm.com
ID: 2GCT:Z5PM:Y3OE:2OLW:AM7W:OKFF:PVMR:72SP:RF4M:LACZ:ZN3E:K5YX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.5"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.5 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.5"
Red Hat Enterprise Linux Server release 7.5 (Maipo)
Red Hat Enterprise Linux Server release 7.5 (Maipo)
zw143 picture zw143 on 15 Jul 2019

杀死异常启动的大量，释放内存缓冲区，重启docker deamon，恢复正常

GeekWolfBoy picture GeekWolfBoy on 23 Jul 2019

I hit this issue on my Kubernetes cluster. The Docker version is v19.3.2. Every time I got this issue, the host can not login like below, so, we should reboot it by using VMware management.

 ~  ssh [email protected]
ssh_exchange_identification: read: Connection reset by peer
 ~  ssh [email protected]
ssh_exchange_identification: read: Connection reset by peer

The cluster's events log below:

longhorn-p-j9wh7    18m         Warning   Unhealthy          pod/engine-image-ei-9bea8a9c-cnf8x                        Readiness probe failed: OCI runtime state failed: fork/exec /usr/bin/runc: cannot allocate memory: : unknown
longhorn-p-j9wh7    18m         Warning   Unhealthy          pod/engine-image-ei-9bea8a9c-cnf8x                        Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "read init-p: connection reset by peer": unknown
longhorn-p-j9wh7    18m         Warning   Unhealthy          pod/engine-image-ei-9bea8a9c-cnf8x                        Readiness probe failed: OCI runtime state failed: exec failed: container_linux.go:345: starting container process caused "read init-p: connection reset by peer": unknown
longhorn-p-j9wh7    18m         Warning   Unhealthy          pod/engine-image-ei-9bea8a9c-cnf8x                        Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:83: starting setns process caused \"fork/exec /proc/self/exe: cannot allocate memory\"": unknown
longhorn-p-j9wh7    3m23s       Warning   Unhealthy          pod/engine-image-ei-9bea8a9c-cnf8x                        Readiness probe failed: OCI runtime state failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:83: starting setns process caused \"fork/exec /proc/self/exe: cannot allocate memory\"": unknown
longhorn-p-j9wh7    19m         Warning   Unhealthy          pod/pvc-79a50cd0-be95-11e9-9163-0050569e4290-e-dac00206   Readiness probe failed: OCI runtime state failed: fork/exec /usr/bin/runc: cannot allocate memory: : unknown
longhorn-p-j9wh7    18m         Warning   Unhealthy          pod/pvc-79a50cd0-be95-11e9-9163-0050569e4290-e-dac00206   Readiness probe failed: OCI runtime exec failed: runc did not terminate sucessfully: unknown
longhorn-p-j9wh7    18m         Warning   Unhealthy          pod/pvc-79a50cd0-be95-11e9-9163-0050569e4290-e-dac00206   Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:83: starting setns process caused \"fork/exec /proc/self/exe: cannot allocate memory\"": unknown
longhorn-p-j9wh7    18m         Warning   Unhealthy          pod/pvc-79a50cd0-be95-11e9-9163-0050569e4290-e-dac00206   Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "read init-p: connection reset by peer": unknown
longhorn-p-j9wh7    4m3s        Warning   Unhealthy          pod/pvc-79a50cd0-be95-11e9-9163-0050569e4290-e-dac00206   Readiness probe failed: OCI runtime state failed: exec failed: container_linux.go:345: starting container process caused "read init-p: connection reset by peer": unknown
longhorn-p-j9wh7    18m         Warning   Unhealthy          pod/pvc-79a50cd0-be95-11e9-9163-0050569e4290-e-dac00206   Readiness probe failed: OCI runtime state failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:83: starting setns process caused \"fork/exec /proc/self/exe: cannot allocate memory\"": unknown
longhorn-p-j9wh7    16m         Normal    Attached           volume/pvc-79a50cd0-be95-11e9-9163-0050569e4290           volume pvc-79a50cd0-be95-11e9-9163-0050569e4290 has been attached to woker5
longhorn-p-j9wh7    11m         Normal    Ready              node/woker5                                               Disk /var/lib/rancher/longhorn/ on node woker5 is ready
longhorn-p-j9wh7    13m         Normal    Schedulable        node/woker5                                               Disk /var/lib/rancher/longhorn/ on node woker5 is schedulable
longhorn-p-j9wh7    13m         Warning   NoDiskInfo         node/woker5                                               Disk /var/lib/rancher/longhorn/ on node woker5 is not ready: Get disk information error: cannot get disk info of directory /var/lib/rancher/longhorn/: Failed to execute: nsenter [--mount=/host/proc/1404/ns//mnt stat -fc {"path":"%n","fsid":"%i","type":"%T","freeBlock":%f,"totalBlock":%b,"blockSize":%S} /var/lib/rancher/longhorn/], output , stderr, , error fork/exec /usr/bin/nsenter: cannot allocate memory
longhorn-p-j9wh7    13m         Warning   DiskPressure       node/woker5                                               unable to schedule any replica to disk /var/lib/rancher/longhorn/ on node woker5
sigma-demo          16m         Normal    SandboxChanged     pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Pod sandbox changed, it will be killed and re-created.
sigma-demo          16m         Normal    Pulled             pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Container image "busybox:1.25.0" already present on machine
sigma-demo          16m         Normal    Created            pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Created container remove-lost-found
sigma-demo          16m         Normal    Started            pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Started container remove-lost-found
sigma-demo          17m         Normal    Pulled             pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Container image "mysql:5.7.14" already present on machine
sigma-demo          17m         Normal    Created            pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Created container sigma-mysql-mysql
sigma-demo          17m         Normal    Started            pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Started container sigma-mysql-mysql
sigma-demo          18m         Warning   Unhealthy          pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Liveness probe failed: OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "read init-p: connection reset by peer": unknown
sigma-demo          18m         Warning   Unhealthy          pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:91: executing setns process caused \"exit status 27\"": unknown
sigma-demo          18m         Warning   Unhealthy          pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Liveness probe failed: OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:83: starting setns process caused \"fork/exec /proc/self/exe: cannot allocate memory\"": unknown
sigma-demo          18m         Warning   Unhealthy          pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:83: starting setns process caused \"fork/exec /proc/self/exe: cannot allocate memory\"": unknown
sigma-demo          18m         Warning   Unhealthy          pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Liveness probe failed: OCI runtime state failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:83: starting setns process caused \"fork/exec /proc/self/exe: cannot allocate memory\"": unknown
sigma-demo          3m12s       Normal    Killing            pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Container sigma-mysql-mysql failed liveness probe, will be restarted
sigma-demo          18m         Warning   Unhealthy          pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Readiness probe failed: OCI runtime state failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:83: starting setns process caused \"fork/exec /proc/self/exe: cannot allocate memory\"": unknown
sigma-demo          17m         Warning   Unhealthy          pod/sigma-mysql-mysql-6895d8b697-5d7xd                    Liveness probe failed: OCI runtime state failed: runc did not terminate sucessfully: runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f89317cd337 m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f89317cd337
stack: frame={sp:0x7fff52280eb8, fp:0x0} stack=[0x7fff51a82428,0x7fff52281450)
00007fff52280db8:  0000000000000000  0000000000000000
00007fff52280dc8:  0000000000000000  0000000000000000
00007fff52280dd8:  0000000000000000  0000000000000000
00007fff52280de8:  0000000000000000  0000000000000000
00007fff52280df8:  0000000000000000  0000000000000000
00007fff52280e08:  0000000000000000  0000000000000000
00007fff52280e18:  0000000000000000  0000000000000000
00007fff52280e28:  0000000000000000  0000000000000000
00007fff52280e38:  0000000000000000  2e656d69746e7572
00007fff52280e48:  6863676563726f66  0000000000000000
00007fff52280e58:  0000000000000000  656e6961746e6f63
00007fff52280e68:  2f3a6e69622f6472  61636f6c2f727375
00007fff52280e78:  2f3a6e6962732f6c  0000000000000002
00007fff52280e88:  00007f8931b5f868  0000555b3736c10a
00007fff52280e98:  0000555b385692c0  0000000000000011
00007fff52280ea8:  0000555b3735d494  0000000000000000
00007fff52280eb8: <00007f89317cea28  0000000000000020
00007fff52280ec8:  0000000000000000  0000000000000000
00007fff52280ed8:  0000000000000000  0000000000000000
00007fff52280ee8:  0000000000000000  0000000000000000
00007fff52280ef8:  0000000000000000  0000000000000000
00007fff52280f08:  0000000000000000  0000000000000000
00007fff52280f18:  0000000000000000  0000000000000000
00007fff52280f28:  0000000000000000  0000000000000000
00007fff52280f38:  0000000000000000  0000000000000000
00007fff52280f48:  0000000000000000  0000000000000000
00007fff52280f58:  0000000000000000  0000000000000000
00007fff52280f68:  0000000000000000  0000000000000000
00007fff52280f78:  0000000000000000  0000000000000000
00007fff52280f88:  0000000000000000  0000000000000000
00007fff52280f98:  0000000000000000  0000000000000000
00007fff52280fa8:  0000000000000000  0000000000000000
runtime: unknown pc 0x7f89317cd337
stack: frame={sp:0x7fff52280eb8, fp:0x0} stack=[0x7fff51a82428,0x7fff52281450)
00007fff52280db8:  0000000000000000  0000000000000000
00007fff52280dc8:  0000000000000000  0000000000000000
00007fff52280dd8:  0000000000000000  0000000000000000
00007fff52280de8:  0000000000000000  0000000000000000
00007fff52280df8:  0000000000000000  0000000000000000
00007fff52280e08:  0000000000000000  0000000000000000
00007fff52280e18:  0000000000000000  0000000000000000
00007fff52280e28:  0000000000000000  0000000000000000
00007fff52280e38:  0000000000000000  2e656d69746e7572
00007fff52280e48:  6863676563726f66  0000000000000000
00007fff52280e58:  0000000000000000  656e6961746e6f63
00007fff52280e68:  2f3a6e69622f6472  61636f6c2f727375
00007fff52280e78:  2f3a6e6962732f6c  0000000000000002
00007fff52280e88:  00007f8931b5f868  0000555b3736c10a
00007fff52280e98:  0000555b385692c0  0000000000000011
00007fff52280ea8:  0000555b3735d494  0000000000000000
00007fff52280eb8: <00007f89317cea28  0000000000000020
00007fff52280ec8:  0000000000000000  0000000000000000
00007fff52280ed8:  0000000000000000  0000000000000000
00007fff52280ee8:  0000000000000000  0000000000000000
00007fff52280ef8:  0000000000000000  0000000000000000
00007fff52280f08:  0000000000000000  0000000000000000
00007fff52280f18:  0000000000000000  0000000000000000
00007fff52280f28:  0000000000000000  0000000000000000
00007fff52280f38:  0000000000000000  0000000000000000
00007fff52280f48:  0000000000000000  0000000000000000
00007fff52280f58:  0000000000000000  0000000000000000
00007fff52280f68:  0000000000000000  0000000000000000
00007fff52280f78:  0000000000000000  0000000000000000
00007fff52280f88:  0000000000000000  0000000000000000
00007fff52280f98:  0000000000000000  0000000000000000
00007fff52280fa8:  0000000000000000  0000000000000000
Aisuko picture Aisuko on 25 Oct 2019

reinstall
sudo yum remove docker docker-common docker-selinux docker-engine

silverfoxs picture silverfoxs on 14 Nov 2019

There is an article https://success.docker.com/article/error-oci-runtime-create-failed-docker-1809 that may relate to the issue. Finally, I restarted the docker services.

Aisuko picture Aisuko on 18 Nov 2019

I use windows subsystem for Linux (WSL) tool in windows. On giving "docker-compose up:" command in my debian terminal, getting this error -

Cannot start service be: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting

Can some body help me whats wrong ?

MahorShekhar picture MahorShekhar on 30 Jul 2020

Have this problem on Debian Stretch K8's. Unable to mount the volume.

Error:
Cannot start service ####: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting

Docker info

Server Version: 19.03.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog

Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657

Kernel Version: 4.9.0-11-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64

Could somebody advise if there is any fix for this?

rohitrav33ndran picture rohitrav33ndran on 28 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

reallinfo picture reallinfo  ·  7Comments
Ace-Tang picture Ace-Tang  ·  8Comments
jessfraz picture jessfraz  ·  3Comments
Andreyooo picture Andreyooo  ·  4Comments
liron-l picture liron-l  ·  10Comments