Podman: Namespace issue when running unprivileged (non-root) container in archlinux

[clueo8]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Installed podman from pacman, followed guides for /etc/subuid and /etc/subgid and also kernel.unprivileged_userns_clone=1. When trying to do a simple test of running httpd, it fails to run.

Steps to reproduce the issue:

$ podman --log-level=debug run httpd
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver vfs                       
DEBU[0000] Using graph root /home/user/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "vfs"   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/1000]docker.io/library/httpd:latest" 
DEBU[0000] reference "[vfs@/home/user/.local/share/containers/storage+/run/user/1000]docker.io/library/httpd:latest" does not resolve to an image ID 
DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/1000]localhost/httpd:latest" 
DEBU[0000] reference "[vfs@/home/user/.local/share/containers/storage+/run/user/1000]localhost/httpd:latest" does not resolve to an image ID 
DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/1000]docker.io/library/httpd:latest" 
DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/1000]registry.fedoraproject.org/httpd:latest" 
DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/1000]quay.io/httpd:latest" 
DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/1000]registry.access.redhat.com/httpd:latest" 
DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/1000]registry.centos.org/httpd:latest" 
Trying to pull docker.io/library/httpd...
DEBU[0000] reference rewritten from 'docker.io/library/httpd:latest' to 'docker.io/library/httpd:latest' 
DEBU[0000] Trying to pull "docker.io/library/httpd:latest" 
DEBU[0000] Returning credentials from /home/user/.docker/config.json 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for docker.io/library/httpd:latest 
DEBU[0000] error accessing certs directory due to permissions: stat /etc/docker/certs.d/docker.io: permission denied 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] Skipping scan of /etc/docker/certs.d/docker.io due to permission error: open /etc/docker/certs.d/docker.io: permission denied 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?account=usero8&scope=repository%3Alibrary%2Fhttpd%3Apull&service=registry.docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/library/httpd/manifests/latest 
DEBU[0000] Using blob info cache at /home/user/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] Source is a manifest list; copying (only) instance sha256:90cca2f9c32ad25afa180da6b14f35de9990cb02b9007350a5bccef4cac1e1c9 
DEBU[0000] GET https://registry-1.docker.io/v2/library/httpd/manifests/sha256:90cca2f9c32ad25afa180da6b14f35de9990cb02b9007350a5bccef4cac1e1c9 
DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/httpd:latest 
DEBU[0001]  Using default policy section                
DEBU[0001]  Requirement 0: allowed                      
DEBU[0001] Overall: allowed                             
DEBU[0001] Downloading /v2/library/httpd/blobs/sha256:7d85cc3b2d8064182718e70ca9f9601a309bb7499db680e15c3231a0b350a42e 
DEBU[0001] GET https://registry-1.docker.io/v2/library/httpd/blobs/sha256:7d85cc3b2d8064182718e70ca9f9601a309bb7499db680e15c3231a0b350a42e 
Getting image source signatures
DEBU[0001] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0001] ... will first try using the original manifest unmodified 
DEBU[0001] Downloading /v2/library/httpd/blobs/sha256:533f5cf513cb52f93f936a5b55105dd1566e541f85446023a5bb98be505f6b3a 
DEBU[0001] GET https://registry-1.docker.io/v2/library/httpd/blobs/sha256:533f5cf513cb52f93f936a5b55105dd1566e541f85446023a5bb98be505f6b3a 
DEBU[0001] Downloading /v2/library/httpd/blobs/sha256:174a8e3bca83c83d129f5ecf6132af10e1b2948af9900a9df5d7c5585bc135f3 
DEBU[0001] GET https://registry-1.docker.io/v2/library/httpd/blobs/sha256:174a8e3bca83c83d129f5ecf6132af10e1b2948af9900a9df5d7c5585bc135f3 
DEBU[0001] Downloading /v2/library/httpd/blobs/sha256:1ab2bdfe97783562315f98f94c0769b1897a05f7b0395ca1520ebee08666703b 
DEBU[0001] GET https://registry-1.docker.io/v2/library/httpd/blobs/sha256:1ab2bdfe97783562315f98f94c0769b1897a05f7b0395ca1520ebee08666703b 
DEBU[0001] Downloading /v2/library/httpd/blobs/sha256:c8e4c9e948929a74030e044b9346f77177883a8f1de13c37a3deac2608d0c91d 
DEBU[0001] GET https://registry-1.docker.io/v2/library/httpd/blobs/sha256:c8e4c9e948929a74030e044b9346f77177883a8f1de13c37a3deac2608d0c91d 
DEBU[0001] Downloading /v2/library/httpd/blobs/sha256:4568916ecf2d1fa4d380c40d3ba527c2359c1ea910cac4e25c9a9c55025c30a9 
DEBU[0001] GET https://registry-1.docker.io/v2/library/httpd/blobs/sha256:4568916ecf2d1fa4d380c40d3ba527c2359c1ea910cac4e25c9a9c55025c30a9 
DEBU[0001] Detected compression format gzip             
DEBU[0001] Using original blob without modification     
DEBU[0001] Detected compression format gzip             
DEBU[0001] Using original blob without modification     
Copying blob 1ab2bdfe9778 [--------------------------------------] 36.5KiB / 25.8MiB
Copying blob c8e4c9e94892 [--------------------------------------] 485b / 9.9MiB
DEBU[0001] Detected compression format gzip             
DEBU[0001] Using original blob without modification     
DEBU[0001] Detected compression format gzip             
DEBU[0001] Using original blob without modification     
Copying blob 1ab2bdfe9778 done
Copying blob c8e4c9e94892 done
Copying blob 174a8e3bca83 done
Copying blob 4568916ecf2d done
Copying blob 533f5cf513cb done
DEBU[0004] No compression detected                      
DEBU[0004] Using original blob without modification     
Copying config 7d85cc3b2d done
Writing manifest to image destination
Storing signatures
DEBU[0005] Start untar layer                            
ERRO[0005] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument 
DEBU[0005] Error pulling image ref //httpd:latest: Error committing the finished image: error adding layer with blob "sha256:1ab2bdfe97783562315f98f94c0769b1897a05f7b0395ca1520ebee08666703b": ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument 
  ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
Trying to pull registry.fedoraproject.org/httpd...
DEBU[0005] reference rewritten from 'registry.fedoraproject.org/httpd:latest' to 'registry.fedoraproject.org/httpd:latest' 
DEBU[0005] Trying to pull "registry.fedoraproject.org/httpd:latest" 
DEBU[0005] Credentials not found                        
DEBU[0005] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0005]  Using "default-docker" configuration        
DEBU[0005]  No signature storage configuration found for registry.fedoraproject.org/httpd:latest 
DEBU[0005] error accessing certs directory due to permissions: stat /etc/docker/certs.d/registry.fedoraproject.org: permission denied 
DEBU[0005] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fedoraproject.org 
DEBU[0005] Skipping scan of /etc/docker/certs.d/registry.fedoraproject.org due to permission error: open /etc/docker/certs.d/registry.fedoraproject.org: permission denied 
DEBU[0005] GET https://registry.fedoraproject.org/v2/   
DEBU[0005] Ping https://registry.fedoraproject.org/v2/ status 200 
DEBU[0005] GET https://registry.fedoraproject.org/v2/httpd/manifests/latest 
DEBU[0006] Error pulling image ref //registry.fedoraproject.org/httpd:latest: Error initializing source docker://registry.fedoraproject.org/httpd:latest: Error reading manifest latest in registry.fedoraproject.org/httpd: manifest unknown: manifest unknown 
  manifest unknown: manifest unknown
Trying to pull quay.io/httpd...
DEBU[0006] reference rewritten from 'quay.io/httpd:latest' to 'quay.io/httpd:latest' 
DEBU[0006] Trying to pull "quay.io/httpd:latest"        
DEBU[0006] Credentials not found                        
DEBU[0006] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0006]  Using "default-docker" configuration        
DEBU[0006]  No signature storage configuration found for quay.io/httpd:latest 
DEBU[0006] error accessing certs directory due to permissions: stat /etc/docker/certs.d/quay.io: permission denied 
DEBU[0006] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
DEBU[0006] Skipping scan of /etc/docker/certs.d/quay.io due to permission error: open /etc/docker/certs.d/quay.io: permission denied 
DEBU[0006] GET https://quay.io/v2/                      
DEBU[0006] Ping https://quay.io/v2/ status 401          
DEBU[0006] GET https://quay.io/v2/auth?scope=repository%3Ahttpd%3Apull&service=quay.io 
DEBU[0006] Increasing token expiration to: 60 seconds   
DEBU[0006] GET https://quay.io/v2/httpd/manifests/latest 
DEBU[0006] Error pulling image ref //quay.io/httpd:latest: Error initializing source docker://quay.io/httpd:latest: Error reading manifest latest in quay.io/httpd: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n" 
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/httpd...
DEBU[0006] reference rewritten from 'registry.access.redhat.com/httpd:latest' to 'registry.access.redhat.com/httpd:latest' 
DEBU[0006] Trying to pull "registry.access.redhat.com/httpd:latest" 
DEBU[0006] Credentials not found                        
DEBU[0006] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0006]  Using "default-docker" configuration        
DEBU[0006]  No signature storage configuration found for registry.access.redhat.com/httpd:latest 
DEBU[0006] error accessing certs directory due to permissions: stat /etc/docker/certs.d/registry.access.redhat.com: permission denied 
DEBU[0006] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com 
DEBU[0006] Skipping scan of /etc/docker/certs.d/registry.access.redhat.com due to permission error: open /etc/docker/certs.d/registry.access.redhat.com: permission denied 
DEBU[0006] GET https://registry.access.redhat.com/v2/   
DEBU[0006] Ping https://registry.access.redhat.com/v2/ status 200 
DEBU[0006] GET https://registry.access.redhat.com/v2/httpd/manifests/latest 
DEBU[0007] Error pulling image ref //registry.access.redhat.com/httpd:latest: Error initializing source docker://registry.access.redhat.com/httpd:latest: Error reading manifest latest in registry.access.redhat.com/httpd: name unknown: Repo not found 
  name unknown: Repo not found
Trying to pull registry.centos.org/httpd...
DEBU[0007] reference rewritten from 'registry.centos.org/httpd:latest' to 'registry.centos.org/httpd:latest' 
DEBU[0007] Trying to pull "registry.centos.org/httpd:latest" 
DEBU[0007] Credentials not found                        
DEBU[0007] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0007]  Using "default-docker" configuration        
DEBU[0007]  No signature storage configuration found for registry.centos.org/httpd:latest 
DEBU[0007] error accessing certs directory due to permissions: stat /etc/docker/certs.d/registry.centos.org: permission denied 
DEBU[0007] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.centos.org 
DEBU[0007] Skipping scan of /etc/docker/certs.d/registry.centos.org due to permission error: open /etc/docker/certs.d/registry.centos.org: permission denied 
DEBU[0007] GET https://registry.centos.org/v2/          
DEBU[0007] Ping https://registry.centos.org/v2/ status 200 
DEBU[0007] GET https://registry.centos.org/v2/httpd/manifests/latest 
DEBU[0007] Error pulling image ref //registry.centos.org/httpd:latest: Error initializing source docker://registry.centos.org/httpd:latest: Error reading manifest latest in registry.centos.org/httpd: manifest unknown: manifest unknown 
  manifest unknown: manifest unknown
ERRO[0007] unable to pull httpd: 5 errors occurred:
    * Error committing the finished image: error adding layer with blob "sha256:1ab2bdfe97783562315f98f94c0769b1897a05f7b0395ca1520ebee08666703b": ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
    * Error initializing source docker://registry.fedoraproject.org/httpd:latest: Error reading manifest latest in registry.fedoraproject.org/httpd: manifest unknown: manifest unknown
    * Error initializing source docker://quay.io/httpd:latest: Error reading manifest latest in quay.io/httpd: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
    * Error initializing source docker://registry.access.redhat.com/httpd:latest: Error reading manifest latest in registry.access.redhat.com/httpd: name unknown: Repo not found
    * Error initializing source docker://registry.centos.org/httpd:latest: Error reading manifest latest in registry.centos.org/httpd: manifest unknown: manifest unknown

Describe the results you received:

ERRO[0005] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument

Describe the results you expected:
The container should run without error.

Additional information you deem important (e.g. issue happens only occasionally):

$ grep user /etc/sub*
/etc/subgid:user:1000000:65536
/etc/subuid:user:1000000:65536

$ cat /etc/sysctl.d/userns.conf
kernel.unprivileged_userns_clone=1

Output of podman version:

$ podman version
Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.12.8
OS/Arch:            linux/amd64

Output of podman info --debug:

$ podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.8
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.0, commit: e217fdff82e0b1a6184a28c43043a4065083407f'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 836374528
  MemTotal: 16690532352
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: archlinux
  kernel: 4.19.67-1-lts
  os: linux
  rootless: true
  uptime: 99h 30m 28.54s (Approximately 4.12 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/user/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/user/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /run/user/1000
  VolumePath: /home/user/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

$ pacman -Q podman
podman 1.5.1-1

Additional environment details (AWS, VirtualBox, physical, etc.):
physical

[rhatdan]

podman unshare cat /proc/self/uid_map

[clueo8]

$ cat /proc/self/uid_map
         0          0 4294967295
$ podman unshare cat /proc/self/uid_map
         0       1000          1
$ podman run httpd
Trying to pull docker.io/library/httpd...
Getting image source signatures
Copying blob c8e4c9e94892 done
Copying blob 174a8e3bca83 done
Copying blob 1ab2bdfe9778 done
Copying blob 4568916ecf2d done
Copying blob 533f5cf513cb done
Copying config 7d85cc3b2d done
Writing manifest to image destination
Storing signatures
ERRO[0005] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument 
  ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
Trying to pull registry.fedoraproject.org/httpd...
  manifest unknown: manifest unknown
Trying to pull quay.io/httpd...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/httpd...
  name unknown: Repo not found
Trying to pull registry.centos.org/httpd...
  manifest unknown: manifest unknown
Error: unable to pull httpd: 5 errors occurred:
    * Error committing the finished image: error adding layer with blob "sha256:1ab2bdfe97783562315f98f94c0769b1897a05f7b0395ca1520ebee08666703b": ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
    * Error initializing source docker://registry.fedoraproject.org/httpd:latest: Error reading manifest latest in registry.fedoraproject.org/httpd: manifest unknown: manifest unknown
    * Error initializing source docker://quay.io/httpd:latest: Error reading manifest latest in quay.io/httpd: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
    * Error initializing source docker://registry.access.redhat.com/httpd:latest: Error reading manifest latest in registry.access.redhat.com/httpd: name unknown: Repo not found
    * Error initializing source docker://registry.centos.org/httpd:latest: Error reading manifest latest in registry.centos.org/httpd: manifest unknown: manifest unknown

[rhatdan]

podman unshare cat /proc/self/uid_map
0 1000 1

This indicates that you are not running in a user namespace with more then one UID. So either /etc/subuid is setup incorrectly or newuidmap and newgidmap are not working correctly.

[rhatdan]

Is their an entry for the user who is UID 1000, inside of /etc/subuid and /etc/subgid?

[clueo8]

$ id
uid=1000(user) gid=1000(user) groups=1000(user)
$ grep user /etc/subuid
user:1000000:65536
$ grep user /etc/subgid
user:1000000:65536

[rhatdan]

Weird.
Does newuidmap and newgidmap executables exist on your system and are they setuid or at least setfcap.

[rhatdan]

#  filecap /usr/bin/newuidmap
set       file                 capabilities
effective /usr/bin/newuidmap     setuid

[clueo8]

See below:

# whereis newuidmap
newuidmap: /usr/bin/newuidmap /usr/share/man/man1/newuidmap.1.gz
# whereis newgidmap
newgidmap: /usr/bin/newgidmap /usr/share/man/man1/newgidmap.1.gz

# filecap /usr/bin/newuidmap
file                 capabilities
/usr/bin/newuidmap     setuid

# filecap /usr/bin/newgidmap
file                 capabilities
/usr/bin/newgidmap     setgid

[vrothberg]

Are they owned by root?

[clueo8]

Yes:

# ls -l /usr/bin/new{uid,gid}*
-rwxr-xr-x 1 root root 41088 Jul 31 15:12 /usr/bin/newgidmap
-rwxr-xr-x 1 root root 36992 Jul 31 15:12 /usr/bin/newuidmap

[clueo8]

I was able to reproduce this issue in a fresh/separate Archlinux, all steps listed below:

sudo pacman -Syu
sudo pacman -S podman
sudo sysctl kernel.unprivileged_userns_clone=1
sudo su
echo 'kernel.unprivileged_userns_clone=1' > /etc/sysctl.d/userns.conf
echo "myUser:1000000:65536" >> /etc/subuid                                    
echo "myUser:1000000:65536" >> /etc/subgid
exit
podman run httpd
...
ERRO[0005] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument 
  ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
...

[mheon]

I think we need to look into better debug information for setting up the user namespace - there's nothing in the log-level=debug output saying what failed here.

[rhatdan]

@clueo8 Could you try this experiment to see if newuidmap is working on archlinux?

https://unix.stackexchange.com/questions/450081/first-process-in-a-new-linux-user-namespace-needs-to-call-setuid

[clueo8]

Let me know how this looks, these namespaces are new to me...

First terminal:

[user@archlinux ~]$ PS1='% ' unshare -U bash
[nobody@archlinux ~]$ echo $$
19566
[nobody@archlinux ~]$ id
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)

Second terminal:

[user@archlinux ~]$ ps -p 19566 -o uid
  UID
 1000
[user@archlinux ~]$ echo '500000 1000 1' >/proc/19566/uid_map 
[user@archlinux ~]$ ps -p 19566 -o uid
  UID
 1000

First terminal:

[nobody@archlinux ~]$ id
uid=500000 gid=65534(nobody) groups=65534(nobody)

[rhatdan]

Looks like user namespace is working. Not sure why podman unshare is not.
@giuseppe @nalind Any ideas?

[clueo8]

I rebooted my server and now it appears to be working!

[vrothberg]

Great. I assume the machine wasn't rebooted after kernel.unprivileged_userns_clone=1?

[clueo8]

Correct. I did both sudo sysctl kernel.unprivileged_userns_clone=1 and echo 'kernel.unprivileged_userns_clone=1' > /etc/sysctl.d/userns.conf and no reboot until this morning. I thought the first command was able to make this change during the current session?

[vrothberg]

I'd expected it to work via sysctl. @rhatdan might know?

[rhatdan]

No idea, I would have thought so also, but maybe it is just for new logins? Or maybe a reboot is required.

[rhatdan]

Closing since it now works.

[clueo8]

I did try logging in with a new session. I'm checking my pacman logs and it looks like my kernel was upgraded that morning and I did not reboot after that...

pacman.log:

[2019-08-27 07:00] [ALPM] upgraded linux-lts (4.19.67-1 -> 4.19.68-1)
[2019-08-27 07:00] [ALPM] upgraded linux-lts-headers (4.19.67-1 -> 4.19.68-1)

podman info from above still had the old version: kernel: 4.19.67-1-lts

The same thing happened in my test arch system, pacman -Syu upgraded the kernel right before running sysctl:

[2019-08-27 09:00] [ALPM] upgraded linux-lts (4.19.66-1 -> 4.19.68-1)
[2019-08-27 09:00] [ALPM] upgraded linux-lts-headers (4.19.66-1 -> 4.19.68-1)

Lesson learned, always reboot after kernel upgrades.