Podman: Unable to pull images without root

[Fogapod]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman gives error when I try to pull images. This does not happen when I use same command as root.

Steps to reproduce the issue:

1. podman pull alpine

Describe the results you received:

Trying to pull docker.io/library/alpine...
Getting image source signatures
Copying blob 9d48c3bd43c5 done
Copying config 9617696764 done
Writing manifest to image destination
Storing signatures
ERRO[0008] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/shadow): lchown /etc/shadow: invalid argument 
  ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/shadow): lchown /etc/shadow: invalid argument

Describe the results you expected:
Image to be pulled without errors.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.12.8
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.8
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.0, commit: e217fdff82e0b1a6184a28c43043a4065083407f'
  Distribution:
    distribution: manjaro
    version: unknown
  MemFree: 1951240192
  MemTotal: 8240787456
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree: 8055353344
  SwapTotal: 8468295680
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: laptop
  kernel: 5.2.11-1-MANJARO
  os: linux
  rootless: true
  uptime: 73h 53m 24.35s (Approximately 3.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/eugene/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/eugene/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /run/user/1000
  VolumePath: /home/eugene/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

$ pacman -Qi podman
Name            : podman
Version         : 1.5.1-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/libpod
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  device-mapper  iptables  libseccomp  ostree  runc  skopeo  btrfs-progs  slirp4netns  libsystemd
Optional Deps   : podman-docker: for Docker-compatible CLI
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 102.71 MiB
Packager        : Bartłomiej Piotrowski <[email protected]>
Build Date      : Sat 17 Aug 2019 11:16:49 PM MSK
Install Date    : Sat 21 Sep 2019 04:38:31 PM MSK
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Additional environment details (AWS, VirtualBox, physical, etc.):

[Fogapod]

Found a related issue: #3890 , but doing sysctl kernel.unprivileged_userns_clone=1 && echo 'kernel.unprivileged_userns_clone=1' > /etc/sysctl.d/userns.conf did not help (with reboot)

additional information:

$ cat /proc/self/uid_map
         0          0 4294967295

$ podman unshare cat /proc/self/uid_map
         0       1000          1


$ whereis newuidmap 
newuidmap: /usr/bin/newuidmap /usr/share/man/man1/newuidmap.1.gz

$ whereis newgidmap
newgidmap: /usr/bin/newgidmap /usr/share/man/man1/newgidmap.1.gz

[mheon]

Can you cat /etc/sub*id and include the output?

[Fogapod]

$ cat /etc/sub*id      
eugene:1000000:65536
eugene:1000000:65536

[rhatdan]

Try removing all of the content from your homedir.

rm -rf ~/.config/containers ~/.local/share/containers
podman system migrate
podman unshare cat /proc/self/uid_map

[Fogapod]

This solved the problem for me. I am able to pull images again.

$ rm -rf ~/.config/containers ~/.local/share/containers
podman system migrate
podman unshare cat /proc/self/uid_map
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
         0       1000          1
         1    1000000      65536

[ivzhh]

In the code pkg/rootless/rootless_linux.go:225, if the newuidmap is missing, running podman system migrate won't solve the "there might not be enough IDs available in the namespace" error. Because the error of cannot find newuidmap is discarded here.

    if uids != nil {
        err := tryMappingTool("newuidmap", pid, os.Geteuid(), uids)
        uidsMapped = err == nil
    }

[rhatdan]

If newuidmap is not available then rootless podman is not going to work, or at least it will not work with more then one UID.

[cmdrogogov]

Apologies for re-opening -

This issue appears to manifest when you're attempting to utilize podman with a domain account even if its in sudoers.

I am assuming this is down to the following;
https://github.com/containers/podman/blob/master/rootless.md