podman on silverblue fails to create uid namespace

Created on 16 Sep 2019 · 9Comments · Source: containers/podman

/kind bug

Description

The current setup on Fedora Silverblue doesn't work for me.

Steps to reproduce the issue:

First, install Fedora 31 Silverblue. Then:

$ podman run --rm -it fedora:latest
Trying to pull docker.io/library/fedora:latest...
Getting image source signatures
Copying blob 5a915a173fbc done
Copying config e9ed59d2ba done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
Trying to pull registry.fedoraproject.org/fedora:latest...
  Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
Trying to pull quay.io/fedora:latest...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/fedora:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/fedora:latest...
  manifest unknown: manifest unknown
Error: unable to pull fedora:latest: 5 errors occurred:
    * Error committing the finished image: error adding layer with blob "sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
    * Error initializing source docker://registry.fedoraproject.org/fedora:latest: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
    * Error initializing source docker://quay.io/fedora:latest: Error reading manifest latest in quay.io/fedora: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
    * Error initializing source docker://registry.access.redhat.com/fedora:latest: Error reading manifest latest in registry.access.redhat.com/fedora: name unknown: Repo not found
    * Error initializing source docker://registry.centos.org/fedora:latest: Error reading manifest latest in registry.centos.org/fedora: manifest unknown: manifest unknown

Output of podman version:

$ podman version
Version:            1.5.1-dev
RemoteAPI Version:  1
Go Version:         go1.13rc1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13rc1
  podman version: 1.5.1-dev
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: 118fcdfca36d706f766bad2663b11bd2c41bf2e7'
  Distribution:
    distribution: fedora
    version: "31"
  MemFree: 10766471168
  MemTotal: 16370573312
  OCIRuntime:
    package: crun-0.8-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun 0.8
      spec: 1.0.0
      +SYSTEMD +SELINUX +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8266969088
  SwapTotal: 8266969088
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: localhost.localdomain
  kernel: 5.3.0-0.rc6.git0.1.fc31.x86_64
  os: linux
  rootless: true
  uptime: 43m 0.32s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/nmccallu/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/nmccallu/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 0
  RunRoot: /run/user/16827
  VolumePath: /var/home/nmccallu/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

$ cat /etc/sub*id
nmccallu:10000:65536
nmccallu:10000:65536

$ podman unshare cat /proc/self/uid_map
         0      16827          1

$ getenforce 
Permissive

kinbug rootless

Source

npmccallum

All 9 comments

@giuseppe PTAL

mheon on 16 Sep 2019

could you try?

podman system migrate && podman unshare cat /proc/self/uid_map

If you still see the same error, what is the output for getcap /usr/bin/newuidmap /usr/bin/newgidmap?

giuseppe on 17 Sep 2019

I see that you also got:

    * Error initializing source docker://registry.fedoraproject.org/fedora:latest: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message

While it's not the main point of this issue, you can set the GODEBUG environment variable to tls13=0 to work around it until the Fedora registry is fixed to work with Go's TLS 1.3 implementation.

debarshiray on 17 Sep 2019

@giuseppe

$ podman system migrate && podman unshare cat /proc/self/uid_map
WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids 
         0      16827          1

$ podman run --rm -it fedora:latest
Trying to pull docker.io/library/fedora:latest...
Getting image source signatures
Copying blob 5a915a173fbc done
Copying config e9ed59d2ba done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
Trying to pull registry.fedoraproject.org/fedora:latest...
  Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
Trying to pull quay.io/fedora:latest...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/fedora:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/fedora:latest...
  manifest unknown: manifest unknown
Error: unable to pull fedora:latest: 5 errors occurred:
    * Error committing the finished image: error adding layer with blob "sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
    * Error initializing source docker://registry.fedoraproject.org/fedora:latest: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
    * Error initializing source docker://quay.io/fedora:latest: Error reading manifest latest in quay.io/fedora: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
    * Error initializing source docker://registry.access.redhat.com/fedora:latest: Error reading manifest latest in registry.access.redhat.com/fedora: name unknown: Repo not found
    * Error initializing source docker://registry.centos.org/fedora:latest: Error reading manifest latest in registry.centos.org/fedora: manifest unknown: manifest unknown

$ getcap /usr/bin/newuidmap /usr/bin/newgidmap
/usr/bin/newuidmap = cap_setuid+ep
/usr/bin/newgidmap = cap_setgid+ep

npmccallum on 17 Sep 2019

@debarshiray

$ GODEBUG=tls13=0 podman run --rm -it fedora:latest
Trying to pull docker.io/library/fedora:latest...
Getting image source signatures
Copying blob 5a915a173fbc done
Copying config e9ed59d2ba done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob ed60cb1abc2e done
Copying config 02781e9f50 done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:22 for /run/utmp): lchown /run/utmp: invalid argument
Trying to pull quay.io/fedora:latest...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/fedora:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/fedora:latest...
  manifest unknown: manifest unknown
Error: unable to pull fedora:latest: 5 errors occurred:
    * Error committing the finished image: error adding layer with blob "sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
    * Error committing the finished image: error adding layer with blob "sha256:ed60cb1abc2e112aa7a26e9f52cc0982551f7e959441522260e03f47437d42b9": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:22 for /run/utmp): lchown /run/utmp: invalid argument
    * Error initializing source docker://quay.io/fedora:latest: Error reading manifest latest in quay.io/fedora: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
    * Error initializing source docker://registry.access.redhat.com/fedora:latest: Error reading manifest latest in registry.access.redhat.com/fedora: name unknown: Repo not found
    * Error initializing source docker://registry.centos.org/fedora:latest: Error reading manifest latest in registry.centos.org/fedora: manifest unknown: manifest unknown

npmccallum on 17 Sep 2019

The problem was that my user's UID was in the middle of the sub?id range. After fixing this, another problem was revealed. I have submitted it here: https://bugzilla.redhat.com/show_bug.cgi?id=1753328

npmccallum on 18 Sep 2019

👍1

@debarshiray

$ GODEBUG=tls13=0 podman run --rm -it fedora:latest
Trying to pull docker.io/library/fedora:latest...
Getting image source signatures
Copying blob 5a915a173fbc done
Copying config e9ed59d2ba done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob ed60cb1abc2e done
Copying config 02781e9f50 done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:22 for /run/utmp): lchown /run/utmp: invalid argument
Trying to pull quay.io/fedora:latest...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/fedora:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/fedora:latest...
  manifest unknown: manifest unknown
Error: unable to pull fedora:latest: 5 errors occurred:
  * Error committing the finished image: error adding layer with blob "sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
  * Error committing the finished image: error adding layer with blob "sha256:ed60cb1abc2e112aa7a26e9f52cc0982551f7e959441522260e03f47437d42b9": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:22 for /run/utmp): lchown /run/utmp: invalid argument
  * Error initializing source docker://quay.io/fedora:latest: Error reading manifest latest in quay.io/fedora: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
  * Error initializing source docker://registry.access.redhat.com/fedora:latest: Error reading manifest latest in registry.access.redhat.com/fedora: name unknown: Repo not found
  * Error initializing source docker://registry.centos.org/fedora:latest: Error reading manifest latest in registry.centos.org/fedora: manifest unknown: manifest unknown

The error from registry.fedoraproject.org is gone now. :)

debarshiray on 18 Sep 2019

The problem was that my user's UID was in the middle of the sub?id range.

That sounds a bit like https://github.com/debarshiray/toolbox/issues/268

Do you remember what the exact problem was? eg., what was your UID and what the subuid range looked like? I am trying to figure out if we can have a better error message for cases like these.

debarshiray on 24 Sep 2019

@debarshiray I just noticed your comment here. The problem is that my UID came from sssd. So I didn't even think about what it was. A good check would be to see if the user's UID is in the middle of their subuid range and warn them. In general, I think a warning about how to set up a proper /etc/subuid could be useful.

This is partly the problem that the kernel doesn't provide any real allocation strategy for UID. We need essentially protected mode for UID/GID where login agents can reserve ranges for their use. Then, after login, user agents can request ranges. But someone has to own this allocation strategy.

As it stands right now, I believe there are vulnerabilities happening today where UID collisions across namespaces allow for escalation of privleges and accidental data disclosure.

npmccallum on 12 Feb 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings