Podman: Ubuntu 19.10 package conflicts with kubeadm

[jcassee]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When trying to install version 1.8.0~7 of podman on Ubuntu 19.10, apt fails with the following error:

You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 podman : Depends: containernetworking-plugins (>= 0.8.1) but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

Running apt-get install -f results in:

The following additional packages will be installed:
  containernetworking-plugins
The following NEW packages will be installed:
  containernetworking-plugins
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
1 not fully installed or removed.
Need to get 0 B/8.454 kB of archives.
After this operation, 52,2 MB of additional disk space will be used.
dpkg: regarding .../containernetworking-plugins_0.8.5~1_amd64.deb containing containernetworking-plugins:
 containernetworking-plugins conflicts with kubernetes-cni
  kubernetes-cni (version 0.7.5-00) is present and installed.

dpkg: error processing archive /var/cache/apt/archives/containernetworking-plugins_0.8.5~1_amd64.deb (--unpack):
 conflicting packages - not installing containernetworking-plugins
Errors were encountered while processing:
 /var/cache/apt/archives/containernetworking-plugins_0.8.5~1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

The package kubernetes-cni is installed because of kubadm.

Steps to reproduce the issue:

1. Set up Ubuntu 19.10 to install podman.

2. Try to install podman.

Describe the results you received:

Podman 0.8.5 cannot be installed.

Describe the results you expected:

Podman 0.8.5 can be installed.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

ersion:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.12.10
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.10
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v1
  Conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.10, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "19.10"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 950431744
  MemTotal: 16667303936
  OCIRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
      spec: 1.0.1-dev
  SwapFree: 376832
  SwapTotal: 1027600384
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: stevens
  kernel: 5.3.0-40-generic
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: 'slirp4netns: /usr/bin/slirp4netns'
    Version: |-
      slirp4netns version 0.4.3
      commit: unknown
  uptime: 8h 5m 19.46s (Approximately 0.33 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  ConfigFile: /home/joost/.config/containers/storage.conf
  ContainerStore:
    number: 4
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  GraphRoot: /home/joost/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 4
  RunRoot: /run/user/1000/containers
  VolumePath: /home/joost/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

Listing...
podman/unknown 1.8.0~7 amd64 [upgradable from: 1.8.0~6]
podman/unknown 1.8.0~7 arm64
podman/unknown 1.8.0~7 armhf

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

@baude @lsm5 Alright, I guess this is what happens when we turn CNI from Recommends to Requires. Can we move the plugins install directory, maybe?

[baude]

i dont see kubernetes-cni ini the official package repo so it too must be something that is in a separate repo like us. if we go back to the soft requires, then we could end up with backlevel cni code (pre 0.8) which would be really bad. ideally we could share the same cni package.

[jcassee]

@baude Good point. I installed kubeadm (and so kubernetes-cni) from the Kubernetes APT repository (apt.kubernetes.io):

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

[adamcavendish]

Hi I wonder whether there're ways to intall an old version of podman. Since I need to create new VMs and it's breaking my kubernetes installation, is it possible for me to switch back to an old version?

[jayaddison]

Seconding @jcassee's report here that this conflict is reproducible when both the upstream kubernetes.io apt source (xenial) and kubic apt source (eoan) are enabled.

The same error was reported by dpkg -- containernetworking-plugins conflicts with kubernetes-cni.

[lsm5]

@jcassee @jayaddison I pushed a new build to OBS just now which has a podman-rootless package which doesn't depend on containernetworking-plugins but is otherwise the exact same as the podman package. Should be available in a bit. It should be available already on the OBS testing repo in case anyone would like to try it out, https://build.opensuse.org/project/show/devel:kubic:libcontainers:testing

[jayaddison]

Thanks @lsm5 , I'll look out for that. I'd imagine podman-rootless will be a temporary package name until the conflict between containernetworking-plugins and kubernetes-cni is resolved?

[lsm5]

Thanks @lsm5 , I'll look out for that. I'd imagine podman-rootless will be a temporary package name until the conflict between containernetworking-plugins and kubernetes-cni is resolved?

ummmmmm, I guess that's here to stay unless maybe OBS and kube's repos are merged or something, and I don't wanna spend any of my own time on that. But if any volunteers wanna look into it, I'd be glad to help out wherever possible.

[lsm5]

I'll PR on podman.io's installation docs to mention this package as well for kube users.

[jcassee]

@lsm5 Thought the podman-rootless packages would be available by now, but it does not seem to contain binaries:

$ dpkg -L podman-rootless
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/podman-rootless
/usr/share/doc/podman-rootless/changelog.gz
/usr/share/doc/podman-rootless/copyright

[lsm5]

@jcassee err sorry, fixing..

[lsm5]

@jcassee will be fixed in 1.8.2~144 https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman . For the previous build, I forgot to explicitly list the files in each package which caused the issue you mentioned.

[lsm5]

The build is available now. I'll close this issue. Please re-open if it persists.

[jayaddison]

This issue persists with podman:1.8.2~144:

 $ sudo apt info kubeadm
Package: kubeadm
Version: 1.18.1-00
Priority: optional
Section: misc
Maintainer: Kubernetes Authors <[email protected]>
Installed-Size: 39.8 MB
Depends: kubelet (>= 1.13.0), kubectl (>= 1.13.0), kubernetes-cni (>= 0.7.5), cri-tools (>= 1.13.0)
Homepage: https://kubernetes.io
Download-Size: 8,163 kB
APT-Manual-Installed: yes
APT-Sources: http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
Description: Kubernetes Cluster Bootstrapping Tool
 The Kubernetes command line tool for bootstrapping a Kubernetes cluster.

N: There are 145 additional records. Please use the '-a' switch to see them.
 $ sudo apt info podman
Package: podman
Version: 1.8.2~144
Priority: optional
Section: devel
Maintainer: Lokesh Mandvekar <[email protected]>
Installed-Size: 88.7 MB
Depends: libseccomp2, libdevmapper1.02.1, libgpgme11, catatonit, conmon (>= 1.0.0), containers-common (>= 0.1.40-9), containernetworking-plugins (>= 0.8.1), cri-o-runc, iptables, podman-plugins
Recommends: slirp4netns, uidmap, varlink
Conflicts: podman-rootless
Homepage: https://github.com/containers/libpod.git
Download-Size: 19.5 MB
APT-Manual-Installed: yes
APT-Sources: http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10  Packages
Description: Manage pods, containers and container images.

 $ sudo apt upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 podman : Depends: containernetworking-plugins (>= 0.8.1) but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
 $ 

[lsm5]

You'll need to uninstall podman and install podman-rootless.

[jayaddison]

Ok, thanks @lsm5!

That's confirmed - podman-rootless:1.8.2~144 from Kubic can co-exist with kubernetes-cni:0.7.5-00 from Kubernetes.io without install conflicts.

Addressing the conflict in the original podman package may still be worthwhile -- it'd significantly ease the migration path from Project Atomic's PPA, leading to better adoption of the Kubic OBS PPA.

(and less important but still relevant: the package name podman-rootless could be confusing as it might lead users to think it's required for use of rootless container mode).

Is there another way we could avoid kubernetes-cni and containernetworking-plugins from conflicting in the first place? As far as I can tell, conflicts are generally caused by overlapping filenames.

[jcassee]

@lsm5 Sorry for the late reply. Thanks for adding the podman-rootless package. Using it now alongside kubeadm.

[lsm5]

@jayaddison ..

Addressing the conflict in the original podman package may still be worthwhile -- it'd significantly ease the migration path from Project Atomic's PPA, leading to better adoption of the Kubic OBS PPA.

IIRC, the package in the projectatomic PPA used to depend on containernetworking-plugins too. So, I don't see how the podman package in OBS is different from the one in the PPA, dependency wise atleast. Could you give more details about how migration is affected atm?

(and less important but still relevant: the package name podman-rootless could be confusing as it might lead users to think it's required for use of rootless container mode).

@mheon @baude is podman-no-cni (or something which makes it obvious) a better name for podman-rootless ?

Is there another way we could avoid kubernetes-cni and containernetworking-plugins from conflicting in the first place? As far as I can tell, conflicts are generally caused by overlapping filenames.

I guess what we could do is not provide the -plugins package ourselves but depend on the kube apt repo for it. But I wonder if that might create other problems other than reduce our freedom to modify/fix packages as we see fit.
@mheon @baude wdyt?

[jayaddison]

Replying inline, and re-ordered:

I guess what we could do is not provide the -plugins package ourselves but depend on the kube apt repo for it. But I wonder if that might create other problems other than reduce our freedom to modify/fix packages as we see fit.

Agree with you on avoiding causing additional problems -- the old phrase 'first, do no harm' springs to mind.

So, I don't see how the podman package in OBS is different from the one in the PPA, dependency wise atleast. Could you give more details about how migration is affected atm?

During system changes, especially ones affecting low-level infrastructure and that may have limited staffing, it's good to consider those staff and make their operations as safe and predictable as possible (while still requiring clear, manual commands from them).

The smoothest migration from ProjectAtomic to Kubic on an apt-based system would involve a system administrator adding the Kubic PPA and then running apt update && apt upgrade. The package versions should help apt know which packages to install.

Although podman is just one package, the identifier podman exists not just for apt install ... commands but also likely appears in dependency lists - some of which are in private infrastructure. We can best serve all users (even if we can't inspect their systems) by keeping component identifiers consistent, even if that means it takes longer to set up and configure the packages.

IIRC, the package in the projectatomic PPA used to depend on containernetworking-plugins too.

The current version of podman in projectatomic _recommends_ containernetworking-plugins but doesn't _require_ it:

$ sudo apt show podman
Package: podman
Version: 1.6.2-1~ubuntu19.04~ppa1
...
Depends: libseccomp2, libdevmapper1.02.1, libgpgme11, conmon (>= 1.0.0), containers-common (>= 0.1.37), cri-o-runc
Recommends: slirp4netns, containernetworking-plugins (>= 0.8.1), uidmap
...
APT-Sources: http://ppa.launchpad.net/projectatomic/ppa/ubuntu disco/main amd64 Packages
Description: Manage pods, containers and container images.

That means that podman can be installed at the same time as kubernetes-cni, as long as containernetworking-plugins isn't. Someone might have been referring to that earlier in the comments.

[fvma01]

So we are forced to use podman-rootless which has some shortcomings:
https://github.com/containers/podman/blob/master/rootless.md

A fix or a valid workaround would be welcome.

$ podman --version
podman version 2.1.1
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:47:53Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

~$ sudo apt show podman
Package: podman
Version: 2.1.1~2
Priority: optional
Section: devel
Maintainer: Lokesh Mandvekar <[email protected]>
Installed-Size: 68.5 MB
Depends: libseccomp2, libdevmapper1.02.1, libgpgme11, catatonit, conmon (>= 2.0.18~1), containers-common (>= 1.1.1~2), containernetworking-plugins (>= 0.8.6~1), runc | cri-o-runc, iptables, podman-plugins
Recommends: crun, slirp4netns, uidmap, varlink
Conflicts: podman-rootless
Homepage: https://github.com/containers/libpod.git
Download-Size: 16.6 MB
APT-Manual-Installed: yes
APT-Sources: http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04  Packages
Description: Manage pods, containers and container images.

$ sudo apt update -qq && sudo apt install -qq -y kubelet kubeadm kubectl
All packages are up to date.
The following additional packages will be installed:
  conntrack cri-tools ebtables kubernetes-cni socat
Suggested packages:
  nftables
The following NEW packages will be installed:
  conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/64.8 MB of archives.
After this operation, 288 MB of additional disk space will be used.
Selecting previously unselected package conntrack.
(Reading database ... 79748 files and directories currently installed.)
Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ...
Unpacking conntrack (1:1.4.5-2) ...
Selecting previously unselected package cri-tools.
Preparing to unpack .../1-cri-tools_1.17.0~3_amd64.deb ...
Unpacking cri-tools (1.17.0~3) ...
Selecting previously unselected package ebtables.
Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ...
Unpacking ebtables (2.0.11-3build1) ...
dpkg: regarding .../3-kubernetes-cni_0.8.7-00_amd64.deb containing kubernetes-cni:
 containernetworking-plugins conflicts with kubernetes-cni
  kubernetes-cni (version 0.8.7-00) is to be installed.

dpkg: error processing archive /tmp/apt-dpkg-install-rVTQN9/3-kubernetes-cni_0.8.7-00_amd64.deb (--unpack):
 conflicting packages - not installing kubernetes-cni
Selecting previously unselected package socat.
Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ...
Unpacking socat (1.7.3.3-2) ...
Selecting previously unselected package kubelet.
Preparing to unpack .../5-kubelet_1.19.3-00_amd64.deb ...
Unpacking kubelet (1.19.3-00) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../6-kubectl_1.19.3-00_amd64.deb ...
Unpacking kubectl (1.19.3-00) ...
Selecting previously unselected package kubeadm.
Preparing to unpack .../7-kubeadm_1.19.3-00_amd64.deb ...
Unpacking kubeadm (1.19.3-00) ...
Errors were encountered while processing:
 /tmp/apt-dpkg-install-rVTQN9/3-kubernetes-cni_0.8.7-00_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

[mheon]

@lsm5 Is the podman-rootless package still available?

[fvma01]

Used some brute force to get them both installed....

sudo apt update -qq && sudo apt install -qq -y podman kubelet kubeadm kubectl 
cp -a /var/cache/apt/archives/containernetworking-plugins_0.8.7~1_amd64.deb /tmp/
mkdir container
dpkg-deb -R containernetworking-plugins_0.8.7~1_amd64.deb container/
sed -i -e '/^Version:/s/$/~conflictfree/' -e '/^Conflicts: kubernetes-cni/d' container/DEBIAN/control
rm -f container/opt/cni/bin/*
sudo dpkg -b container/ containernetworking-plugins_0.8.7~1_amd64_custom.deb
sudo dpkg -i containernetworking-plugins_0.8.7~1_amd64_custom.deb
sudo apt update -qq && sudo apt install -qq -y podman kubelet kubeadm kubectl 
sudo apt-mark hold kubelet kubeadm kubectl podman containernetworking-plugins

$ dpkg -s kubernetes-cni
Package: kubernetes-cni
Status: install ok installed
Priority: optional
Section: misc
Installed-Size: 70475
Maintainer: Kubernetes Authors <[email protected]>
Architecture: amd64
Version: 0.8.7-00
Description: Kubernetes CNI
 The binaries required to provision container networking
Homepage: https://kubernetes.io

$ dpkg -s containernetworking-plugins
Package: containernetworking-plugins
Status: install ok installed
Priority: extra
Section: devel
Installed-Size: 53028
Maintainer: Lokesh Mandvekar <[email protected]>
Architecture: amd64
Version: 0.8.7~1~conflictfree
Depends: libc6 (>= 2.4)
Description: Libraries for writing CNI plugins
 The CNI (Container Network Interface) project consists of a specification
 and libraries for writing plugins to configure network interfaces in Linux
 containers, along with a number of supported plugins. CNI concerns itself
 only with network connectivity of containers and removing allocated resources
 when the container is deleted.
Built-Using: golang-1.13 (= 1.13.8-1ubuntu1)
Homepage: https://github.com/containernetworking/plugins

$ ls -l /opt/cni/bin/
total 70496
-rwxr-xr-x 1 root root  4159518 May 13 19:50 bandwidth
-rwxr-xr-x 1 root root  4671647 May 13 19:50 bridge
-rwxr-xr-x 1 root root 12124326 May 13 19:50 dhcp
-rwxr-xr-x 1 root root  5945760 May 13 19:50 firewall
-rwxr-xr-x 1 root root  3069556 May 13 19:50 flannel
-rwxr-xr-x 1 root root  4174394 May 13 19:50 host-device
-rwxr-xr-x 1 root root  3614480 May 13 19:50 host-local
-rwxr-xr-x 1 root root  4314598 May 13 19:50 ipvlan
-rwxr-xr-x 1 root root  3209463 May 13 19:50 loopback
-rwxr-xr-x 1 root root  4389622 May 13 19:50 macvlan
-rwxr-xr-x 1 root root  3939867 May 13 19:50 portmap
-rwxr-xr-x 1 root root  4590277 May 13 19:50 ptp
-rwxr-xr-x 1 root root  3392826 May 13 19:50 sbr
-rwxr-xr-x 1 root root  2885430 May 13 19:50 static
-rwxr-xr-x 1 root root  3356587 May 13 19:50 tuning
-rwxr-xr-x 1 root root  4314446 May 13 19:50 vlan

[pascal71]

containernetworking-plugins_0.9.0~1_amd64.deb has the same issues.
Need any help in resolving it?

[zhangguanzhang]

@lsm5 PTAL