podman failing in Travis CI again, now with write /proc/self/attr/keycreate: invalid argument

[adelton]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Attempt to run podman build on Travis CI's Ubuntu Xenial fails with

container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"write /proc/self/attr/keycreate: invalid argument\"

Steps to reproduce the issue:

1. In .travis.yml, install podman using sudo add-apt-repository -y ppa:projectatomic/ppa && sudo apt-get update -y && sudo apt-get install -y podman
2. Run sudo podman build for some Dockerfile, for example using https://github.com/adelton/freeipa-container/tree/podman

Describe the results you received:

https://travis-ci.org/adelton/freeipa-container/jobs/512956513 failed with

STEP 4: RUN groupadd -g 288 kdcproxy ; useradd -u 288 -g 288 -c 'IPA KDC Proxy User' -d '/var/lib/kdcproxy' -s '/sbin/nologin' kdcproxy
error running container: error creating container for [/bin/sh -c groupadd -g 288 kdcproxy ; useradd -u 288 -g 288 -c 'IPA KDC Proxy User' -d '/var/lib/kdcproxy' -s '/sbin/nologin' kdcproxy]: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"write /proc/self/attr/keycreate: invalid argument\"

Describe the results you expected:

The same build not failing, for example https://travis-ci.org/adelton/freeipa-container/jobs/508818334.

Additional information you deem important (e.g. issue happens only occasionally):

It is a regression.

Output of podman version:

No way to run podman version but the Travis CI log says

Unpacking cri-o-runc (1.0.0-rc6-1~ubuntu16.04.2~ppa74) ...
Selecting previously unselected package podman.
Preparing to unpack .../podman_1.2.0-1~ubuntu16.04.2~ppa14_amd64.deb ...
Unpacking podman (1.2.0-1~ubuntu16.04.2~ppa14) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Setting up conmon (0-1~dev~ubuntu16.04~ppa2) ...
Setting up containers-golang (0.1-1~dev~ubuntu16.04~ppa1) ...
Setting up containers-common (0.1.36-1~dev~ubuntu16.04.2~ppa9) ...
Setting up libgpgme11:amd64 (1.6.0-1) ...
Setting up containernetworking-plugins (0.7.3-1~ubuntu16.04.2~ppa2) ...
Setting up cri-o-runc (1.0.0-rc6-1~ubuntu16.04.2~ppa74) ...
Setting up podman (1.2.0-1~ubuntu16.04.2~ppa14) ...

Output of podman info --debug:

N/A

Additional environment details (AWS, VirtualBox, physical, etc.):

Travis CI.

[rhatdan]

The updated runc package is causing this problem.

[giuseppe]

is this still happening? It is fixed in upstream runc

[adelton]

Yes, the situation has improved: https://travis-ci.org/adelton/freeipa-container/jobs/512956506.

[giuseppe]

@adelton could we close this issue?

[adelton]

Could we use this as an opportunity to add some automated testing of podman setup/behaviour Travis CI, so that any potential regressions are caught and reported as soon as possible?

[giuseppe]

we stopped using travis for podman testing as the kernel there is too old and stuff like rootless user namespaces require a newer one. Do you have anything in particular in mind on what we could test more? We could probably add the test to the cirrus CI we are using right now

[adelton]

I want to help podman by testing freeipa-container not just with docker https://github.com/freeipa/freeipa-container/blob/master/.travis.yml but with podman as well: https://github.com/adelton/freeipa-container/blob/podman/.travis.yml. For my purposes Travis CI works fine.
But from time to time, the podman installation and behaviour on Xenial which I'm using on Travis CI gets broken. And I'd prefer for the podman team catching it early.

[rhatdan]

There is an updated runc that should fix this problem.

[marshallford]

I'm seeing this issue in the Fedora 29 official container image after a dnf install -y podman on GKE.

[root@podman /]# cat Dockerfile 
FROM debian:jessie
RUN echo hello world
[root@podman /]# podman build -t test:latest .
STEP 1: FROM debian:jessie
STEP 2: RUN echo hello world
container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"write /proc/self/attr/keycreate: invalid argument\""
error running container: error creating container for [/bin/sh -c echo hello world]: : exit status 1
Error: error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[echo hello world] Flags:[] Attrs:map[] Message:RUN echo hello world Original:RUN echo hello world}: error while running runtime: exit status 1
[root@podman /]# podman version
Version:            1.2.0
RemoteAPI Version:  1
Go Version:         go1.11.5
OS/Arch:            linux/amd64
[root@podman /]# runc -v
runc version 1.0.0-rc6+dev
commit: 1d3f73d4086533a858613bc4b6af2b5e882f4730
spec: 1.0.1-dev

apiVersion: v1
kind: Pod
metadata:
  name: podman
  namespace: default
spec:
  containers:
  - name: fedora
    image: fedora:29
    securityContext:
      privileged: true
    command:
      - cat
    tty: true
    volumeMounts:
    - name: podman
      mountPath: /var/lib/containers
  restartPolicy: Always
  volumes:
  - name: podman
    hostPath:
      path: /var/lib/containers

[rhatdan]

Is SELinux enabled on your system?

[marshallford]

TLDR: No.

After trying getenforce, cat /etc/sysconfig/selinux, and sestatus I'm fairly confident that SELinux isn't even an option with Google's Container-Optimized OS with containerd, which is the node image for the GKE cluster that I have been testing with. Based on my naive findings via the K8s docs the host OS must have SELinux enabled in order to set options within the container.

[rhatdan]

There is an updated version of runc that was just committed to upstream.
@lsm5 Do we ship runc? If yes then this fix should be in tomorrows repo.

Otherwise you could downgrade to the previous version of runc to remove the breakage,

[marshallford]

I've tried runc version 1.0.0-rc7 as well and get the same error as above. Are you referring to an upcoming 1.0.0-rc8 release?

[rhatdan]

Yes, I got the fix merged this morning, and they were waiting for the fix for rc8/

[marshallford]

I've confirmed that 1.0.0-rc8 fixes the issue I listed above. I will have to look into Fedora's packaging process and/or symlink runc to the the downloaded binary. @rhatdan Thank you for all of your help!

[rhatdan]

runc should be available in updates-testing and be released by early next week.

[gbraad]

No updates available on F29 yet

[marshallford]

@gbraad Same for F30. Just checked.

[rhatdan]

According to https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc70b381ad it should be in stable now. Well it reached that state yesterday.

[gbraad]

@rhatdan how about F29? I am not willing to update the whole system at this moment as I will present soon. I guess the advisory update is available as sudo dnf upgrade --advisory=FEDORA-2019-6174b47003

---

confirmed to work

[rhatdan]

It should be there as well.

If it has not reached a repo yet, then just install it from updates-testing.

dnf -y update runc --enablerepo=updates-testing.