Podman: Running rootless containers on ubuntu bionic doesn't work

[emonty]

kind bug

Description

After having installed podman on an ubuntu bionic system using the PPA and attempting to run a container, I got:

* Error committing the finished image: error adding layer with blob "sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f": ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument

The machine in question is a recently installed from scratch Ubuntu Bionic with not a ton installed.

Steps to reproduce the issue:

1. Install podman:

sudo add-apt-repository ppa:projectatomic/ppa
sudo apt-get update
sudo apt-get install podman

1. Install a registries.conf file

# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: 'registries.search', 'registries.insecure', 
# and 'registries.block'.

[registries.search]
registries = ['docker.io']

# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = []


# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []

1. Try to run a container:

podman run -it --rm ubuntu /bin/sh

Describe the results you received:

Trying to pull docker.io/ubuntu:latest...Getting image source signatures
Copying blob sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f
 30.62 MB / 30.62 MB [======================================================] 7s
Copying blob sha256:da1315cffa03c17988ae5c66f56d5f50517652a622afc1611a8bdd6c00b1fde3
 847 B / 847 B [============================================================] 0s
Copying blob sha256:fa83472a3562898caaf8d77542181a473a84039376f2ba56254619d9317ba00d
 556 B / 556 B [============================================================] 0s
Copying blob sha256:f85999a86bef2603a9e9a4fa488a7c1f82e471cbb76c3b5068e54e1a9320964a
 162 B / 162 B [============================================================] 0s
Copying config sha256:93fd78260bd1495afb484371928661f63e64be306b7ac48e2d13ce9422dfee26
 3.32 KB / 3.32 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
ERRO[0011] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument 
Failed
unable to pull ubuntu: 1 error occurred:

* Error committing the finished image: error adding layer with blob "sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f": ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument

Describe the results you expected:

To be in a shell in the container.

Additional information you deem important (e.g. issue happens only occasionally):

Happens for other containers - it's not just an ubuntu container.

Output of podman version:

Version:       0.11.2-dev
Go Version:    go1.10.4
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.5-dev
  Conmon:
    package: 'cri-o-1.12: /usr/lib/crio/bin/conmon'
    path: /usr/lib/crio/bin/conmon
    version: 'conmon version 1.12.4-dev, commit: '
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 190324736
  MemTotal: 8241233920
  OCIRuntime:
    package: 'cri-o-runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 483840000
  SwapTotal: 1027600384
  arch: amd64
  cpus: 4
  hostname: solace
  kernel: 4.15.0-29-generic
  os: linux
  rootless: true
  uptime: 73h 46m 19.84s (Approximately 3.04 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
store:
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: []
  GraphRoot: /home/mordred/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /run/user/1001

Additional environment details (AWS, VirtualBox, physical, etc.):

Lenovo Thinkpad Laptop

[mheon]

Is this running Podman without root?

[emonty]

Yes - totally just running as a normal user.

I went and tried with root and it works. (yay!) So I suppose rootless
podman is the thing that isn't working.

I have my user listed in /etc/subuid and /etc/subgid as per the docs.

On Fri, Nov 30, 2018 at 3:06 PM Matthew Heon [e-mail]
wrote:

Is this running Podman without root?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/1908#issuecomment-443339368,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAFztD5BUoHiCE5zk6liWm1uX6-hE8-hks5u0Z2_gaJpZM4Y8YCY
.

[mheon]

@emonty Can you check if the uidmap package is installed (the newuidmap binary should be installed in /usr/bin if so)

[rhatdan]

What entry do you have in /etc/subuid and /etc/subgid?
This could be attempting to add an UID/GID that is not in the range of your usernamespace.

[emonty]

@mheon BOOM. There it is. I installed that and it worked. That's super exciting!

Perhaps uidmap should go into the Depends for the package? I'll go comment on the packaging issue

[emonty]

For completeness, the following is what is needed to get rootless podman working on ubuntu bionic:

sudo add-apt-repository ppa:projectatomic/ppa
sudo apt-get update
sudo apt-get install podman uidmap
echo "$(whoami):10000:65536" | sudo tee /etc/subuid
echo "$(whoami):10000:65536" | sudo tee /etc/subgid
echo -e "[registries.search]\nregistries = ['docker.io']" | sudo tee /etc/containers/registries.conf

[mheon]

@lsm5 We might want to make uidmap a dependency of Podman in the PPA, so we can have rootless work out of the box

[giuseppe]

this PR should avoid the kind of isses you've reported: https://github.com/containers/libpod/pull/1923

[asm582]

@giuseppe I think I have configured steps mentioned above I still get the error:

abhi@f12n12:~$ podman run -it ubuntu bash
Trying to pull docker.io/library/ubuntu...Getting image source signatures
Copying blob f12d17d7baa6 done
Copying blob fce39298a87b done
Copying blob 404b73f949d7 done
Copying blob cecce6095d35 done
Copying config 4e6d115184 done
Writing manifest to image destination
Storing signatures
ERRO[0004] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted
ERRO[0004] Error pulling image ref //ubuntu:latest: Error committing the finished image: error adding layer with blob "sha256:cecce6095d35c15ae2a27f2e80fbc0a0806d92aa38fbd27d004a9168f6b3aa58": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted
Failed
Error: unable to pull ubuntu: 1 error occurred:
* Error committing the finished image: error adding layer with blob "sha256:cecce6095d35c15ae2a27f2e80fbc0a0806d92aa38fbd27d004a9168f6b3aa58": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted

can you please help?

Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

abhi@f12n12:~$ podman --version
podman version 1.4.4

[giuseppe]

@asm582 can you try to do podman system migrate first? If it doesn't still work, please attach the output of: podman unshare cat /proc/self/uid_map

[asm582]

@giuseppe below is my output:

asmalvan@f12n12:~$ podman unshare cat /proc/self/uid_map
0 2287 1
1 100000 65536

[giuseppe]

can be AppArmor blocking lchown?

[asm582]

@giuseppe I am not sure, Sorry I missed sharing this information. I am running podman on NFS with root squash enabled, should we take this discussion on a git new issue?

[giuseppe]

ah then that is the issue. NFS cannot be used for rootless containers: https://github.com/containers/libpod/blob/master/rootless.md (look for NFS in the page)

[mheon]

Should we add a troubleshooting note that moving directories out of /home will solve this?

[TomSweeneyRedHat]

I'm in the midst of updating the troubleshooting guide for the overlayfs build woes, I'll tag this on too.

[rhatdan]

Maybe we should update containers/storage overlay checking to check if the upper layer is NFS, and the ignorechownerrors flag is not turned on return and error.

[eoli3n]

Found this : https://www.redhat.com/sysadmin/rootless-podman-nfs
It works, but you need to edit user config, as system config is not templated

[rhatdan]

Yes we have a PR for this in container storage, but have not been able to get back to it.

https://github.com/containers/storage/pull/518