Podman: Ubuntu 18.04 PPA support

[ipmb]

kind bug

Description

I'm installing podman from the PPA on Ubuntu 18.04 and seeing the same issue described in https://github.com/containers/libpod/issues/1260.

Running apt install podman pulls in cri-o-1.10 which, based on the discussion in the other issue, appears to be too old and the cause of the error.

Steps to reproduce the issue:

sudo add-apt-repository ppa:projectatomic/ppa
sudo apt-get update
sudo apt-get install podman
echo -e "[registries.search]\nregistries = ['docker.io']" | sudo tee /etc/containers/registries.conf
sudo podman run --rm -it alpine sh

Describe the results you received:

$ sudo podman run --rm -it alpine sh
option parsing failed: Unknown option --log-level
                                                 write child: broken pipe

Describe the results you expected:

Container runs

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:       0.9.4-dev
Go Version:    go1.10
OS/Arch:       linux/amd64

Output of podman info:

host:
  Conmon:
    package: 'cri-o-1.10: /usr/lib/crio/bin/conmon'
    path: /usr/lib/crio/bin/conmon
    version: 'conmon version 1.10.7-dev, commit: '
  MemFree: 380252160
  MemTotal: 2096058368
  OCIRuntime:
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 279207936
  SwapTotal: 1073737728
  arch: amd64
  cpus: 2
  hostname: util-kitchen
  kernel: 4.9.93-linuxkit-aufs
  os: linux
  uptime: 43h 20m 55.85s (Approximately 1.79 days)
insecure registries:
  registries: null
registries:
  registries:
  - docker.io
store:
  ContainerStore:
    number: 11
  GraphDriverName: vfs
  GraphOptions:
  - vfs.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 5
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):

It's running inside a Docker container (for testing)

[TomSweeneyRedHat]

@lsm5 PTAL

[lsm5]

Not all packages are ready for 18.04 yet. See https://launchpad.net/~projectatomic/+archive/ubuntu/ppa/+packages Working on building the rest..

[lsm5]

podman 0.10.2 (from master branch) should now be available on the PPA for 18.04. PTAL: https://launchpad.net/~projectatomic/+archive/ubuntu/ppa/+packages

[emonty]

Sweet! First feedback - it's doing something with /usr/bin/docker which conflicts at least with the docker-ce package from upstream docker. Perhaps it wants a Conflicts: Provides: and Replaces: on docker-ce in the control file?

Unpacking podman (0.10.2-1~ubuntu18.04~ppa2) ...
dpkg: error processing archive /var/cache/apt/archives/podman_0.10.2-1~ubuntu18.04~ppa2_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/docker', which is also in package docker-ce 18.06.1~ce~3-0~ubuntu
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/podman_0.10.2-1~ubuntu18.04~ppa2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

After removing docker-ce, podman installs great.

[emonty]

Trying to run containers is running in to a few issues. with trying to lchown nonexistent files:

mordred@solace:~/src/git.openstack.org/openstack-infra/project-config$ podman run -it --rm fedora /bin/sh
Trying to pull docker.io/fedora:latest...Getting image source signatures
Copying blob sha256:565884f490d9ec697e519c57d55d09e268542ef2c1340fd63262751fa308f047
 82.90 MB / 82.90 MB [=====================================================] 17s
Copying config sha256:c582c1438f27b3775e2534abc82d14974ecb00c2c53161d05ec73a73d35e1235
 2.29 KB / 2.29 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
ERRO[0019] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: lchown /run/systemd/netif: invalid argument 

and

mordred@solace:~/src/git.openstack.org/openstack-infra/project-config$ podman run -it --rm ubuntu:bionic /bin/sh
Trying to pull docker.io/ubuntu:bionic...Getting image source signatures
Copying blob sha256:473ede7ed136b710ab2dd51579af038b7d00fbbf6a1790c6294c93666203c0a6
 30.55 MB / 30.55 MB [======================================================] 9s
Copying blob sha256:c46b5fa4d940569e49988515c1ea0295f56d0a16228d8f854e27613f467ec892
 843 B / 843 B [============================================================] 0s
Copying blob sha256:93ae3df89c92cb1d20e9c09f499e693d3a8a8cef161f7158f7a9a3b5d06e4ef2
 554 B / 554 B [============================================================] 0s
Copying blob sha256:6b1eed27cadec5de8051d56697b0b67527e4076deedceefb41b7b2ea9b900459
 162 B / 162 B [============================================================] 0s
Copying config sha256:ea4c82dcd15a33e3e9c4c37050def20476856a08e59526fbe533cc4e98387e39
 3.32 KB / 3.32 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
ERRO[0012] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument 

[lsm5]

hmm, i thought i added a conflicts: docker, but let me check.

About the second issue, @mheon @baude ptal

[vrothberg]

Looks like a storage issue, adding @nalind to the pairs of eyes.

[lsm5]

I'm just gonna add Conflicts: (and Provides:) docker.io, docker-ce since I don't wanna just replace someone's prior docker installation.

[emonty]

@lsm5 yes - I think that's totally right

[lsm5]

On Mon, Nov 05, 2018 at 05:57:14AM -0800, Monty Taylor wrote:

@lsm5 yes - I think that's totally right

podman - 0.10.2-1~ubuntu18.04~ppa3 should complain about package conflict

HTH.

--
Lokesh
IRC, GitHub: lsm5
GPG: 0xC7C3A0DD
https://keybase.io/lsm5

[vrothberg]

@emonty, could you share the output of podman info and the logs when re-running the command with podman --log-level=debug? I haven't seen the issue before and can't reproduce in my environment(s).

[emonty]

mordred@solace:~/src/git.openstack.org/openstack-infra/project-config$ podman info
host:
  BuildahVersion: 1.5-dev
  Conmon:
    package: 'cri-o-1.12: /usr/lib/crio/bin/conmon'
    path: /usr/lib/crio/bin/conmon
    version: 'conmon version 1.12.1-dev, commit: '
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 356372480
  MemTotal: 8241229824
  OCIRuntime:
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1'
  SwapFree: 8462004224
  SwapTotal: 8464101376
  arch: amd64
  cpus: 4
  hostname: solace
  kernel: 4.15.0-38-generic
  os: linux
  uptime: 22h 48m 14.45s (Approximately 0.92 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
store:
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: []
  GraphRoot: /home/mordred/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /run/user/1001

and

mordred@solace:~/src/git.openstack.org/openstack-infra/project-config$ podman --log-level=debug run -it --rm ubuntu:bionic /bin/sh
INFO[0000] running as rootless                          
DEBU[0000] Not configuring container store              
INFO[0000] Found CNI network crio-bridge (type=bridge) at /etc/cni/net.d/100-crio-bridge.conf 
INFO[0000] Found CNI network 200-loopback.conf (type=loopback) at /etc/cni/net.d/200-loopback.conf 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] Initializing boltdb state at /home/mordred/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Set libpod namespace to ""                   
WARN[0000] AppArmor security is not available in rootless mode 
DEBU[0000] Using bridge netmode                         
INFO[0000] running as rootless                          
DEBU[0000] [graphdriver] trying provided driver "vfs"   
INFO[0000] Found CNI network crio-bridge (type=bridge) at /etc/cni/net.d/100-crio-bridge.conf 
INFO[0000] Found CNI network 200-loopback.conf (type=loopback) at /etc/cni/net.d/200-loopback.conf 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] Initializing boltdb state at /home/mordred/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] parsed reference into "[vfs@/home/mordred/.local/share/containers/storage+/run/user/1001]docker.io/library/ubuntu:bionic" 
DEBU[0000] reference "[vfs@/home/mordred/.local/share/containers/storage+/run/user/1001]docker.io/library/ubuntu:bionic" does not resolve to an image ID 
DEBU[0000] parsed reference into "[vfs@/home/mordred/.local/share/containers/storage+/run/user/1001]localhost/ubuntu:bionic" 
DEBU[0000] reference "[vfs@/home/mordred/.local/share/containers/storage+/run/user/1001]localhost/ubuntu:bionic" does not resolve to an image ID 
DEBU[0000] parsed reference into "[vfs@/home/mordred/.local/share/containers/storage+/run/user/1001]docker.io/library/ubuntu:bionic" 
Trying to pull docker.io/ubuntu:bionic...DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for docker.io/library/ubuntu:bionic 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?account=emonty&scope=repository%3Alibrary%2Fubuntu%3Apull&service=registry.docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/library/ubuntu/manifests/bionic 
DEBU[0001] Source is a manifest list; copying (only) instance sha256:6b9eb699512656fc6ef936ddeb45ab25edcd17ab94901790989f89dbf782344a 
DEBU[0001] GET https://registry-1.docker.io/v2/library/ubuntu/manifests/sha256:6b9eb699512656fc6ef936ddeb45ab25edcd17ab94901790989f89dbf782344a 
DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/ubuntu:bionic 
DEBU[0001]  Using default policy section                
DEBU[0001]  Requirement 0: allowed                      
DEBU[0001] Overall: allowed                             
DEBU[0001] Downloading /v2/library/ubuntu/blobs/sha256:ea4c82dcd15a33e3e9c4c37050def20476856a08e59526fbe533cc4e98387e39 
DEBU[0001] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:ea4c82dcd15a33e3e9c4c37050def20476856a08e59526fbe533cc4e98387e39 
Getting image source signatures
DEBU[0001] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0001] ... will first try using the original manifest unmodified 
Copying blob sha256:473ede7ed136b710ab2dd51579af038b7d00fbbf6a1790c6294c93666203c0a6
DEBU[0001] Downloading /v2/library/ubuntu/blobs/sha256:473ede7ed136b710ab2dd51579af038b7d00fbbf6a1790c6294c93666203c0a6 
DEBU[0001] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:473ede7ed136b710ab2dd51579af038b7d00fbbf6a1790c6294c93666203c0a6 
DEBU[0001] Detected compression format gzip             
 0 B / 30.55 MB [--------------------------------------------------------------]DEBU[0001] Using original blob without modification     
 30.55 MB / 30.55 MB [======================================================] 6s
Copying blob sha256:c46b5fa4d940569e49988515c1ea0295f56d0a16228d8f854e27613f467ec892
DEBU[0008] Downloading /v2/library/ubuntu/blobs/sha256:c46b5fa4d940569e49988515c1ea0295f56d0a16228d8f854e27613f467ec892 
DEBU[0008] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:c46b5fa4d940569e49988515c1ea0295f56d0a16228d8f854e27613f467ec892 
DEBU[0009] Detected compression format gzip             
 0 B / 843 B [-----------------------------------------------------------------]DEBU[0009] Using original blob without modification     
 843 B / 843 B [============================================================] 0s
Copying blob sha256:93ae3df89c92cb1d20e9c09f499e693d3a8a8cef161f7158f7a9a3b5d06e4ef2
DEBU[0009] Downloading /v2/library/ubuntu/blobs/sha256:93ae3df89c92cb1d20e9c09f499e693d3a8a8cef161f7158f7a9a3b5d06e4ef2 
DEBU[0009] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:93ae3df89c92cb1d20e9c09f499e693d3a8a8cef161f7158f7a9a3b5d06e4ef2 
DEBU[0010] Detected compression format gzip             
 0 B / 554 B [-----------------------------------------------------------------]DEBU[0010] Using original blob without modification     
 554 B / 554 B [============================================================] 0s
Copying blob sha256:6b1eed27cadec5de8051d56697b0b67527e4076deedceefb41b7b2ea9b900459
DEBU[0010] Downloading /v2/library/ubuntu/blobs/sha256:6b1eed27cadec5de8051d56697b0b67527e4076deedceefb41b7b2ea9b900459 
DEBU[0010] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:6b1eed27cadec5de8051d56697b0b67527e4076deedceefb41b7b2ea9b900459 
DEBU[0010] Detected compression format gzip             
 0 B / 162 B [-----------------------------------------------------------------]DEBU[0010] Using original blob without modification     
 162 B / 162 B [============================================================] 0s
Copying config sha256:ea4c82dcd15a33e3e9c4c37050def20476856a08e59526fbe533cc4e98387e39
DEBU[0010] No compression detected                      
 0 B / 3.32 KB [---------------------------------------------------------------]DEBU[0010] Using original blob without modification     
 3.32 KB / 3.32 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
DEBU[0010] Start untar layer                            
ERRO[0010] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument 
DEBU[0010] Error pulling image ref //ubuntu:bionic: Error committing the finished image: error adding layer with blob "sha256:473ede7ed136b710ab2dd51579af038b7d00fbbf6a1790c6294c93666203c0a6": ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument 
Failed
ERRO[0010] unable to pull ubuntu:bionic: 1 error occurred:

* Error committing the finished image: error adding layer with blob "sha256:473ede7ed136b710ab2dd51579af038b7d00fbbf6a1790c6294c93666203c0a6": ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument 

[lsm5]

btw, I'll be splitting the /usr/bin/docker symlink into a podman-docker subpackage, so people can keep podman and docker installed together if they like.

[emonty]

@vrothberg oh - I forgot to tag you earlier when I posted that output. my bad.

[vrothberg]

Thanks a lot, @emonty! We'll look into it :)

[durandj]

@lsm5 Any word on when that split you mentioned will happen? I want to play around with Podman in my local work dev environment but don't really want to nuke my Docker install just yet.

[lsm5]

@durandj i'll do it sometime tonight/early tomorrow morning, should be available tomorrow.

[lsm5]

@durandj podman - 0.11.2-1\~ubuntu18.04\~ppa4 is building now. I have removed the docker CLI for now, I had trouble building split packages. But this build should not interfere with your docker installation. HTH.

[durandj]

@lsm5 Awesome! I'll have to give that a try.

[emonty]

@vrothberg got it working with the help of @mheon ... turns out the issue was missing install of the uidgen package. Maybe we should add that to the depends-on? I know it's not strictly required, but it's not a big depend and it sure does make podman super sexy right out of the box.

Where's the packaging source? I'd love to make a PR for that.

[emonty]

Also, should we maybe put a an /etc/containers/registries.conf in containers-common or podman? I also had to do

echo -e "[registries.search]\nregistries = ['docker.io']" | sudo tee /etc/containers/registries.conf

to get things working, which was a little odd ... but totally workable.

[rhatdan]

Yes we should setup a registries.conf with
docker.io, quay.io, and whatever the ubuntu default container registry if it exists.

[rhatdan]

I think adding a dependency on uidgen is fine (Shadow-utils in Fedora land has this so it is automatically installed.) Is there a concept of weak dependencies in APT?

[emonty]

Yeah - you can list it in Recommends: and, at least on Ubuntu, it'll get installed by default, but uninstalling it won't cause podman to be uninstalled. A person can also skip the recommends with --no-install-recommends when they do the install. I think that's likely a good choice for uidgen here (the chances someone REALLY wants to avoid it are low)

If we wanted to be even more soft, uidgen could go into Suggests: ... but I think it's more likely that people will hit the confusion of "why doesn't rootless work for me" if we went that route.

[rhatdan]

Yes recommends makes more sense. Now if we could only get Podman and Buildah into Debian and Ubuntu default repos....

[emonty]

++ to getting them into default repos!

While I'm just saying random words, it would be cool to have both in alpine as well.

Basically - podman is awesome and I want it in all the places. :)

[baude]

i believe we have this satisfied now, re-open if you feel differently and update the issue