Podman: Error pulling image , Error committing the finished image: error adding layer with blob ...

[juansuerogit]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

/kind feature

Description

I have RHEL servers in the 7.x range ( i think they are 7.4 or 7.5 ) that we currently run containers on with docker-compose. Went to Red Hat conference and learned about Podman so want to use Podman in production to help us get away from the big fat deamons and not to run containers as root.

To that end i have created a centos 7.5 VM on my laptop and installed podman.

But I cannot get rootless podman to work to get to images on docker.io or a managed kubernetes service that has their own registry as well as tools that docker login for me.

i dont understand the relationship of podman images local repo and docker images local repo with additional respect to a managed service thats docker login is logged into.

Steps to reproduce the issue:

1.clean Centos 7.5 VM

1. logged into a regular user called "meta" (not root)
2. sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="/boot/vmlinuz-3.10.0-957.5.1.el7.x86_64"
3. sudo yum -y update && sudo yum install -y podman
4. sudo echo 'user.max_user_namespaces=15076' >> /etc/sysctl.conf
5. sudo echo 'meta:100000:65536' >> /etc/subuid
6. sudo echo 'meta:100000:65536' >> /etc/subgid
7. sudo reboot
   9 podman run -v /home/meta/backup:/root/backup -dt docker.io/centos:latest sleep 100

( root less podman )

Describe the results you received:

WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids
Trying to pull docker.io/centos:latest...Getting image source signatures
Copying blob 8ba884070f61 done
Copying config 9f38484d22 done
Writing manifest to image destination
Storing signatures
ERRO[0026] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 0:54 for /run/lock/lockdev): lchown /run/lock/lockdev: invalid argument
ERRO[0026] Error pulling image ref //centos:latest: Error committing the finished image: error adding layer with blob "sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df": ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 0:54 for /run/lock/lockdev): lchown /run/lock/lockdev: invalid argument
Failed
Error: unable to pull docker.io/centos:latest: unable to pull image: Error committing the finished image: error adding layer with blob "sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df": ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 0:54 for /run/lock/lockdev): lchown /run/lock/lockdev: invalid argument

Describe the results you expected:

expected a running pod/container

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.3.2
RemoteAPI Version:  1
Go Version:         go1.10.3
OS/Arch:            linux/amd64

Output of podman info --debug:

WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids 
debug:
  compiler: gc
  git commit: ""
  go version: go1.10.3
  podman version: 1.3.2
host:
  BuildahVersion: 1.8.2
  Conmon:
    package: podman-1.3.2-1.git14fdcd0.el7.centos.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.14.0-dev, commit: e0b5a754190a3c24175944ff64fa7add6c8b0431-dirty'
  Distribution:
    distribution: '"centos"'
    version: "7"
  MemFree: 350928896
  MemTotal: 3973316608
  OCIRuntime:
    package: containerd.io-1.2.5-3.1.el7.x86_64
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  hostname: min0-kube0
  kernel: 3.10.0-957.21.3.el7.x86_64
  os: linux
  rootless: true
  uptime: 1h 1m 24.65s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ConfigFile: /home/meta/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/meta/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /tmp/1000
  VolumePath: /home/meta/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

[debarshiray]

ERRO[0026] Error while applying layer: ApplyLayer exit status 1
stdout: stderr: there might not be enough IDs available in the
namespace (requested 0:54 for /run/lock/lockdev): lchown
/run/lock/lockdev: invalid argument

The podman info --debug output that you pasted below has the reason for this:

WARN[0000] using rootless single mapping into the namespace. This
might break some images. Check /etc/subuid and /etc/subgid for adding
subids

See the podman(1), subgid(5), subuid(5) and usermod(8) manuals for more information. This is caused by users created with older version of shadow-utils. Newer versions take care of updating /etc/subuid and /etc/subgid for you.

As an example, on my host OS, both those files have:

rishi:100000:65536

If your files are currently empty then you can just go with something similar, but if there are prior entries in those files then you'll have to be careful and avoid overlapping entries.

[baude]

@giuseppe is this fixed?

[mheon]

Probably missing newuidmap and newgidmap executables. Install https://copr.fedorainfracloud.org/coprs/vbatts/shadow-utils-newxidmap/ which will be included in 7.7.

[JJ]

I'm experiencing the same problem with buildah. Here's the debug output:

{
    "debug": {
        "buildah version": "1.12.0-dev",
        "compiler": "gc",
        "git commit": "",
        "go version": "go1.10.4"
    },
    "host": {
        "Distribution": {
            "distribution": "ubuntu",
            "version": "16.04"
        },
        "MemTotal": 16572575744,
        "MenFree": 7008686080,
        "SwapFree": 16929255424,
        "SwapTotal": 16929255424,
        "arch": "amd64",
        "cpus": 4,
        "hostname": "bojack",
        "kernel": "4.15.0-70-generic",
        "os": "linux",
        "rootless": true,
        "uptime": "1h 40m 55.62s (Approximately 0.04 days)"
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "vfs",
        "GraphOptions": null,
        "GraphRoot": "/home/jmerelo/.local/share/containers/storage",
        "GraphStatus": {},
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/user/1000"
    }
}

newuidmap and newgdmap are freshly installed. Still, unable to build anything.

[rhatdan]

What happens when you execute
buildah unshare