Podman: Incompatibilities of podman from docker on Travis CI

[junaruga]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Summary

I tried how podman worked on Travis CI comparing it from the benaviors of docker.
Here is the reproducer on my repository.
https://github.com/junaruga/multiarch-podman-experiment
https://github.com/junaruga/multiarch-podman-experiment/blob/master/.travis.yml

Seeing the result of Travis CI,
https://travis-ci.org/junaruga/multiarch-podman-experiment/builds/565807138

There are total 8 test cases for both podman and docker in the Travis page.

docker use cases

1. docker : Passed
2. docker-dockerfile : Passed

podman ideal use cases replaced by above docker use cases

1. podman : Failed
2. podman-dockerfile : Failed

podman use cases to analyze issues

1. podman-own-registries-podman-run : Passed
2. podman-sudo : Failed
3. podman-own-registries-sudo-podman-run : Failed
4. podman-sudo-podman-run-full-image-with-registry : Failed

• My goal is to pass the test cases "3. podman" and "4. podman-dockerfile" that are cases replaced from "1. docker" and "2. docker-dockerfile", basically replaced "docker" with "podman".
• "6. podman-sudo" is the case that I like to see to be fixed in the highest priority.
• "7. podman-own-registries-sudo-podman-run" and "8. podman-sudo-podman-run-full-image-with-registry" are just to share the information.

Suggestions

I have some suggestions seeing the result.

1. Include /etc/containers/registries.conf to the podman deb package to align the initial state with podman RPM on Fedora. Seeing the result of $ dpkg-query -L podman in the Travis log, there is no registries.conf file in it. I want podman deb package to include the file to give podman users similar experiment with docker or with podman Fedora package.

   /etc/containers/registries.conf

   [registries.search]
   registries = ['docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.access.redhat.com', 'registry.centos.org']
   
   [registries.insecure]
   registries = []
   
   [registries.block]
   registries = []
   

2. Seeing "6. podman-sudo", do you know what is the reason of the error? Is there way to avoid this issue as a temporary workflow? https://travis-ci.org/junaruga/multiarch-podman-experiment/jobs/565807144

   $ sudo podman --log-level debug run --rm docker.io/fedora uname -m
   ERRO[0000] error creating libpod runtime: error creating runtime static files directory /var/lib/containers/storage/libpod: mkdir /var/lib/containers: permission denied 
   

3. Seeing "5. podman-own-registries-podman-run", the test case is passed. But there is a error message regardless the command exists with 0 successfully. https://travis-ci.org/junaruga/multiarch-podman-experiment/jobs/565807143 . Is this related to https://github.com/containers/libpod/issues/2794 ?

   ERRO[0010] could not find slirp4netns, the network namespace won't be configured: exec: "slirp4netns": executable file not found in $PATH
   x86_64
   The command "podman run --rm -t fedora uname -m" exited with 0.
   

Steps to reproduce the issue:

1. See the "6 podman-sudo". The case shows the issue simply
   https://travis-ci.org/junaruga/multiarch-podman-experiment/builds/565807138

2. Run below command on the Travis CI environment. ("6 podman-sudo" case)

   $ sudo -E apt-get -yq --no-install-suggests --no-install-recommends $(travis_apt_get_options) install podman
   $ sudo podman --log-level debug run --rm docker.io/fedora uname -m
   

Describe the results you received:

Below command finishes with the error.

$ sudo podman --log-level debug run --rm docker.io/fedora uname -m
ERRO[0000] error creating libpod runtime: error creating runtime static files directory /var/lib/containers/storage/libpod: mkdir /var/lib/containers: permission denied 
The command "sudo podman --log-level debug run --rm docker.io/fedora uname -m" exited with 125.

• The test cases: "3. podman" and "4. podman-dockerfile" are failed.
• The test case "5. podman-own-registries-podman-run" are passed with the error message ERRO[0010] could not find slirp4netns, the network namespace won't be configured: exec: "slirp4netns": executable file not found in $PATH.

Describe the results you expected:

Below command finishes with success.

$ sudo podman --log-level debug run --rm docker.io/fedora uname -m

• The test cases: "3. podman" and "4. podman-dockerfile" are passed.
• The test case "5. podman-own-registries-podman-run" are passed without an error message.

Additional information you deem important (e.g. issue happens only occasionally):

This comes from the test https://github.com/multiarch/qemu-user-static with podman on Travis CI, that I shared on https://github.com/containers/buildah/issues/1590#issuecomment-493755305 in a past tme.

Output of podman version:

$ podman version
Version:            1.4.3
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

$ podman info --debug
WARN[0000] cannot run `loginctl enable-linger` for the current user: exit status 1 
debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.4.3
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 0.3.0, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "16.04"
  MemFree: 6542876672
  MemTotal: 7835996160
  OCIRuntime:
    package: 'cri-o-runc: /usr/bin/runc'
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  hostname: travis-job-ee3fb551-7f6d-46ac-867a-1f7a887fa827
  kernel: 4.15.0-1028-gcp
  os: linux
  rootless: true
  uptime: 1m 4.43s
registries:
  blocked: null
  insecure: null
  search: null
store:
  ConfigFile: /home/travis/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/travis/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /run/user/2000
  VolumePath: /home/travis/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

Travis CI. The OS is Ubuntu xenial.

Here is the podman deb package information.

$ apt-cache show podman
Package: podman
Priority: optional
Section: devel
Installed-Size: 34246
Maintainer: Lokesh Mandvekar <[email protected]>
Architecture: amd64
Version: 1.4.3-1~ubuntu16.04.2~ppa1
Depends: libseccomp2, libdevmapper1.02.1, libgpgme11, containernetworking-plugins (>= 0.7.5), conmon (>= 0-1~dev~ubuntu16.04~ppa1), containers-common (>= 0.1.36-1~dev~ubuntu16.04.2~ppa7), cri-o-runc
Filename: pool/main/p/podman/podman_1.4.3-1~ubuntu16.04.2~ppa1_amd64.deb
Size: 7422446
MD5sum: bf748c157692b83547826897568f9b74
SHA1: 2f2c9e4c8781343638ab3baf075d675922fbdc9f
SHA256: 201b3cac5989306b20034d7f0db4ec9ba667965b4d95bd27c018c4ade76aca8d
Description-en: Manage pods, containers and container images.
Description-md5: b77ba7b65e652815c2243cb728ae77e6

Here is the file list on the podman deb package. To see the full list, you can check the Travis CI log.

$ dpkg-query -L podman
/.
/etc
/etc/cni
/etc/cni/net.d
/etc/cni/net.d/87-podman-bridge.conflist
/usr
...
/usr/share/man/man1/podman-container-mount.1.gz

Thank you.

[rhatdan]

Well first it looks like podman running as root (Well running thorugh sudo) is not allowed to write to /var/lib/containers/storage? Any idea why that is? Can you gain access to the VM and see why this would fail?

[rhatdan]

@mheon @vrothberg PTAL

[vrothberg]

@junaruga, thanks for opening the issue and providing so many details. I will go through the items incrementally.

Include /etc/containers/registries.conf to the podman deb package to align the initial state with podman RPM on Fedora. Seeing the result of $ dpkg-query -L podman in the Travis log, there is no registries.conf file in it.

@lsm5 PTAL

[baude]

what is the origin of the package you are using for podman?

[junaruga]

@rhatdan

Well first it looks like podman running as root (Well running thorugh sudo) is not allowed to write to /var/lib/containers/storage? Any idea why that is? Can you gain access to the VM and see why this would fail?

Sure, I will take a look at "(Well running thorugh sudo) is not allowed to write to /var/lib/containers/storage" too on my repository's different branch keeping the current master branch.

But as this is the result of Travis CI, you can reproduce the issues too on your repository like this.

1. Go to https://github.com/junaruga/multiarch-podman-experiment repository. Then click "Fork" button.
2. Click "Sync account" button at https://travis-ci.org/account/repositories .
3. Go to https://travis-ci.org/your_name/multiarch-podman-experiment , then click "Activate" button to activate your Git repository's Travis CI
4. Then you can create a new branch and push your commit to your repository, and then Travis will start. You can see the Travis log.

@junaruga, thanks for opening the issue and providing so many details. I will go through the items incrementally.

@vrothberg Sure, thanks for checking this!

[mheon]

I'm going to slap on the ubuntu label and tag @lsm5 for the packaging issues

[junaruga]

what is the origin of the package you are using for podman?

You can check the above result of $ apt-cache show podman.

The repository is sourceline: "ppa:projectatomic/ppa". You can check above my repository's .travis.yml.

.traivs.yml

    sources:
      # To install podman and some container tools.
      # https://github.com/containers/libpod/blob/master/install.md
      - sourceline: "ppa:projectatomic/ppa"
    packages:
      # repository: ppa:projectatomic/ppa
      - podman

[vrothberg]

$ sudo podman --log-level debug run --rm docker.io/fedora uname -m
ERRO[0000] error creating libpod runtime: error creating runtime static files directory /var/lib/containers/storage/libpod: mkdir /var/lib/containers: permission denied 
The command "sudo podman --log-level debug run --rm docker.io/fedora uname -m" exited with 125.

That smells like we just don't have the permissions in Travis. We need to point the run and graph root to some other path which we can either do by editing the storage.conf or by setting it on --root and --run-root on the Podman CLI.

[junaruga]

For the "6. podman-sudo" case, I debugged adding below code on my repository's "debug" branch.

       script:
         - podman version
         - podman info --debug
+        - ls /var/lib || true
+        - ls /var/lib/containers || true
+        - sudo mkdir -p /var/lib/containers/storage/libpod
+        - sudo ls -l /var/lib/containers/storage/libpod
         - sudo podman --log-level debug run --rm docker.io/fedora uname -m

Interestingly sudo mkdir -p /var/lib/containers/storage/libpod works.

See https://travis-ci.org/junaruga/multiarch-podman-experiment/jobs/566003406 for detail.

[junaruga]

For /etc/containers/registries.conf, I found additional information.

On my Fedora 30, below file is included in containers-common-0.1.37-0.gite079f9d.fc30.x86_64.

$ rpm -qf /etc/containers/registries.conf
containers-common-0.1.37-0.gite079f9d.fc30.x86_64

And the file includes

[registries.search]
registries = ['docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.access.redhat.com', 'registry.centos.org']

On the Travis's Ubuntu, I checked adding below code in .travis.yml.

   - dpkg-query -L podman
+
+  - apt-cache show containers-common
+  - ls -l /etc/containers/registries.conf
+  - dpkg -S /etc/containers/registries.conf
+  - cat /etc/containers/registries.conf
+
   - docker --version

Then the result is

$ dpkg -S /etc/containers/registries.conf
containers-common: /etc/containers/registries.conf

$ apt-cache show containers-common
Package: containers-common
Source: skopeo
Priority: optional
Section: devel
Installed-Size: 26
Maintainer: Lokesh Mandvekar <[email protected]>
Architecture: all
Version: 0.1.37-1~dev~ubuntu16.04.2~ppa3
Replaces: skopeo-containers
Provides: skopeo-containers
Depends: containers-golang
Conflicts: skopeo-containers
Filename: pool/main/s/skopeo/containers-common_0.1.37-1~dev~ubuntu16.04.2~ppa3_all.deb
Size: 6242
MD5sum: 3c2aab0968e644ecd357b1baebd42d56
SHA1: addd0c438c4741ab829fb736884d54bf631f04e3
SHA256: 5e9c6b24458e5c726efbc84b0f71457bc7c489dad2cef7877be5f5cd9c0b7409
Description-en: Configuration files for working with image signatures.
Description-md5: f43206d962dedbd395e95478774e12fc

But the file's registries.search is empty like this.

$ cat /etc/containers/registries.conf
...
[registries.search]
registries = []
...

See any case of https://travis-ci.org/junaruga/multiarch-podman-experiment/builds/566009939 for detail.

[junaruga]

We need to point the run and graph root to some other path which we can either do by editing the storage.conf or by setting it on --root and --run-root on the Podman CLI.

I replaced /etc/containers/storage.conf below items to directories under user directory.

[storage]
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"

Then the result is like this.

+sudo podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.4.3
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 1.0.0-rc2, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "16.04"
  MemFree: 6489886720
  MemTotal: 7836004352
  OCIRuntime:
    package: 'cri-o-runc: /usr/bin/runc'
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  hostname: travis-job-ff0cc5f2-8904-45e8-b06e-1eeb85c3bee0
  kernel: 4.15.0-1028-gcp
  os: linux
  rootless: false
  uptime: 1m 29.05s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /home/travis/build/junaruga/fedora-workshop-multiarch/root/var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 0
  RunRoot: /home/travis/build/junaruga/fedora-workshop-multiarch/root/var/run/containers/storage
  VolumePath: /home/travis/build/junaruga/fedora-workshop-multiarch/root/var/lib/containers/storage/volumes

Then I could pass some errors, but I got below another error ERRO[0006] error pulling image "multiarch/qemu-user-static": unable to pull multiarch/qemu-user-static: 1 error occurred:.
Do you have any idea to fix this error?

Thanks.

$ sudo podman --log-level debug pull multiarch/qemu-user-static
DEBU[0000] Initializing boltdb state at /home/travis/build/junaruga/fedora-workshop-multiarch/root/var/lib/containers/storage/libpod/bolt_state.db 
...
DEBU[0004] Applying tar in /home/travis/build/junaruga/fedora-workshop-multiarch/root/var/lib/containers/storage/overlay/dfab9b9b625d6408133ee8543d9b9800254f3a8791b4c919b22c08f5498e686a/diff 
ERRO[0006] Error pulling image ref //multiarch/qemu-user-static:latest: Error committing the finished image: error adding layer with blob "sha256:2a09037e38c15de583d55dfde02ac83e88be99ae40cf6c4c0229941083eed92f": Error processing tar file(exit status 1): operation not permitted 
Failed
ERRO[0006] error pulling image "multiarch/qemu-user-static": unable to pull multiarch/qemu-user-static: 1 error occurred:
    * Error committing the finished image: error adding layer with blob "sha256:2a09037e38c15de583d55dfde02ac83e88be99ae40cf6c4c0229941083eed92f": Error processing tar file(exit status 1): operation not permitted

The command "sudo podman --log-level debug pull multiarch/qemu-user-static" failed and exited with 125 during .

See https://travis-ci.org/junaruga/fedora-workshop-multiarch/jobs/567511767 for detail.

[junaruga]

Excuse me, I was reading "ubuntu" tag in issues tickets.
Is someone actually succeeded to run podman (podman build and podman run) on Ubuntu with Travis CI?
https://github.com/containers/libpod/issues?q=is%3Aissue+label%3Aubuntu+is%3Aopen

[junaruga]

Good news!

I was able to do podman build and podman run on Travis CI by creating /etc/containers/registries.conf file. I was able to pass all the cases I expected.
I referred this document https://clouding.io/kb/en/how-to-install-and-use-podman-on-ubuntu-18-04/ .

Here is the way.

.travis.yml

addons:
  apt:
    config:
      retries: true
    update: true
    sources:
      - sourceline: "ppa:projectatomic/ppa"
    packages:
      - podman
before_install:
  - mkdir -p /etc/containers
  - echo -e "[registries.search]\nregistries = ['docker.io']" | sudo tee /etc/containers/registries.conf

The actual examples are here.
https://github.com/junaruga/multiarch-podman-experiment/blob/master/.travis.yml
https://travis-ci.org/junaruga/multiarch-podman-experiment/builds/591095263

podman still shows an error message ERRO[0010] could not find slirp4netns, the network namespace won't be configured: exec: "slirp4netns": executable file not found in $PATH existing with exit status: 0.
See https://travis-ci.org/junaruga/multiarch-podman-experiment/jobs/591095266#L564

Can I send a pull-request to add this content to run podman on Travis CI to below document?
Because it helps users to run podman on Travis CI.
https://github.com/containers/libpod/blob/master/install.md

[vrothberg]

Hi @junaruga, thanks for sharing!

Can I send a pull-request to add this content to run podman on Travis CI to below document?

Absolutely, yes. Contributions are always welcome!

[rhatdan]

@junaruga We would love to have this added to libpod for testing.

[junaruga]

Sure. I love to contribute it.

[junaruga]

Just memo for me. I found how to install podman on Ubuntu bionic. In the article, the configuration files are set like this.

https://www.n0r1sk.com/post/2019-10-02-podman-with-vxlan-overlay-network-deep-dive/

> mkdir -p /etc/containers
> curl https://raw.githubusercontent.com/projectatomic/registries/master/registries.fedora -o /etc/containers/registries.conf
> curl https://raw.githubusercontent.com/containers/skopeo/master/default-policy.json -o /etc/containers/policy.json

[junaruga]

Just memo for me. rpmlint project is trying to replace docker with podman.
https://github.com/rpm-software-management/rpmlint/pull/351

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[rhatdan]

@junaruga Any progress?

[junaruga]

@rhatdan sorry for the late response. I just sent a pull-request to update the "Installation - Ubuntu" document. For the testing through Cirrus I want, related to your comment https://github.com/containers/libpod/pull/4160#issuecomment-537254738 , no progress so far.

[vrothberg]

Friendly ping. Any updates on this issue?

[junaruga]

I keep in mind and remember the task. But I am sorry. No progress so far.

[rhatdan]

@junaruga We are taking a run through the issues, to get updates on each one.

[cathay4t]

I tried to run dbus and udevd in container of travis using podman.

Without root, the systemd-udevd will never start up as /sys is not writable.
With root, podman failed with:

DEBU[0000] Network is already cleaned up, skipping...   

DEBU[0000] Container
863f70664b18930f09c1454a3d5fc875e5ea0b3342acc7104311e581879f6690 storage is
already unmounted, skipping... 

DEBU[0000] ExitCode msg: "error configuring network namespace for container
863f70664b18930f09c1454a3d5fc875e5ea0b3342acc7104311e581879f6690: missing cni
default network" 

ERRO[0000] error configuring network namespace for container
863f70664b18930f09c1454a3d5fc875e5ea0b3342acc7104311e581879f6690: Missing CNI
default network

[vrothberg]

@cathay4t, which version of Podman are you using and on which distribution (and repository)?

[cathay4t]

@cathay4t, which version of Podman are you using and on which distribution (and repository)?

It's Travis Ubuntu 18.04 with podman version 1.8.0 using repo from https://podman.io/getting-started/installation.html

[vrothberg]

WARN[0000] Error validating CNI config file /etc/cni/net.d/87-podman-bridge.conflist: [failed to find plugin "bridge" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]] 

@lsm5, have you seen this before on 18:04? Looks like some plugins are missing?

[lsm5]

WARN[0000] Error validating CNI config file /etc/cni/net.d/87-podman-bridge.conflist: [failed to find plugin "bridge" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]] 

@lsm5, have you seen this before on 18:04? Looks like some plugins are missing?

Haven't tested podman on ubuntu beyond an 'apt install'. I'll check it now..

[lsm5]

$ apt list podman containernetworking-plugins
Listing... Done
containernetworking-plugins/unknown,now 0.8.5~1 amd64 [installed,automatic]
podman/unknown,now 1.8.0~5 amd64 [installed]

$ ls /opt/cni/bin
bandwidth  bridge  dhcp  firewall  flannel  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sbr  static  tuning  vlan

@vrothberg So, the plugins are present in the expected path.

[cathay4t]

Aha. The containernetworking-plugins is just suggested by podman, not depend on.

I will try to manually install it and try again.

[cathay4t]

@lsm5 Once manually install containernetworking-plugins in .travis.yml. It works like charm.

Maybe a document saying podman as root in ubuntu need containernetworking-plugins could helps others.

Thank you for fixing this.

[manics]

Travis defaults to --no-install-recommends. I think sudo apt-get -qq -y install --install-recommends podman should install everything.

[mheon]

Per discussion, it sounds like we're going to swap it back to a Requires. I remember it being swapped originally so rootless Podman did not need to install root-only dependencies, but if that results in broken installations, it doesn't seem to be worth it.

[lsm5]

This will be fixed in 1.8.0~6 which is currently being uploaded to OBS.

[lsm5]

1.8.0~6 sent to OBS. Should be available soon. Closing...