Podman: Trying to use --init fails with "container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory"

[jlebon]

/kind bug

Description

Running the fedora:31 image with --init fails with:

Creating f31: Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

Steps to reproduce the issue:

1. sudo podman run -ti --privileged --rm --init registry.fedoraproject.org/fedora:31

Describe the results you received:

Creating f31: Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

Describe the results you expected:

No errors

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-3.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 322580480
  MemTotal: 12489740288
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 19308523520
  SwapTotal: 19323154432
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: lux
  kernel: 5.2.17-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 24h 21m 26.53s (Approximately 1.00 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /var/home/jlebon/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/jlebon/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 0
  RunRoot: /tmp/1000
  VolumePath: /var/home/jlebon/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-1.5.1-3.fc30.x86_64

And indeed the catatonic binary is missing from the package:

$ rpm -ql podman | grep libexec
/usr/libexec/podman
/usr/libexec/podman/conmon

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora Silverblue 30.

This used to work, so I'm guessing it was dropped in a recent version.

[jlebon]

Really the actual image used here doesn't matter. I updated the issue title to reflect this.

[rhatdan]

@QiWang19 This is a field that will need to be in containers.conf.

[afbjorklund]

Typical, it seems that I missed the mark after the constant was moved a bit.

sed -e 's|/usr/libexec/podman/catatonit|/usr/local/lib/podman/catatonit|' -i libpod.conf
sed -e 's|/usr/libexec/podman/catatonit|/usr/local/lib/podman/catatonit|' -i libpod/runtime.go

Unfortunately the first line only changed the commented out default config...

#init_path = "/usr/local/lib/podman/catatonit"
Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

Will fix it next time around.

At least I included the binary :-)

tc@box:~$ /usr/local/lib/podman/catatonit --version
tini version 0.1.3_catatonit

[vrothberg]

The problem is that catatonit is not (yet?) packaged for Fedora. Maybe, we should default to tini and require that in the .spec? @rhatdan @lsm5 WDYT?

[rhatdan]

Should be a recommends, and sure we should default to a executable that is shipped in Fedora.

[lsm5]

The problem is that catatonit is not (yet?) packaged for Fedora. Maybe, we should default to tini and require that in the .spec? @rhatdan @lsm5 WDYT?

Well, doesn't look like tini is packaged either. It was bundled inside the docker package but never a standalone one.

[vrothberg]

Is it part of moby?

[lsm5]

Is it part of moby?

Yes, https://src.fedoraproject.org/rpms/moby-engine/blob/master/f/moby-engine.spec#_26 . Will need to be packaged separately.

[lsm5]

I guess I can take that up, unless you prefer we do catatonit itself.

[vrothberg]

I'll ping you on slack to chat :)

[lsm5]

If we can afford to wait a few days, we can get catatonit packaged into Fedora and have that as a Recommends. If you want it by tomorrow itself, I can get it subpackaged for the time being, but there'll be some unnecessary issues around versioning/epoch when the standalone package goes out.

[jlebon]

It's been broken that way for a while now (I only just got around to reporting it now :) ). If catatonic is the upstream default and what gets tested in CI, then my personal recommendation is to just do it properly and package it, even if it takes a few days.

[vrothberg]

I vote for a dedicated catatonit package. Thanks, @lsm5!

@cyphar, FYI :+1:

[cyphar]

I might take a look at requesting that Docker switch to catatonit, which will ensure it's packaged everywhere.

[komish]

Not sure if it's in scope for _this_ issue, but running into the same on CentOS 8

container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

[mheon]

@lsm5 @baude @rhatdan Poke - what do we want to do about this? Substantially harder to get a new package into Cent than Fedora. Do we need to bundle it into the Podman RPM?

[afbjorklund]

Do we need to bundle it into the Podman RPM?

FWIW, Docker is doing this already, as docker-init

$ docker-init --version
tini version 0.18.0 - git.fec3683

Probably fair to bundle stuff under libexec/podman, no ?

[rhatdan]

We should only bundle it, if it was specific to podman.

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[mheon]

Where do we stand here?

[lsm5]

Where do we stand here?

err not yet, I'll try to get this in soon as I can find the time, I really don't want to bundle it though if it has its own versions.

[ruckc]

I'd volunteer doing the work to package this... but i don't know where to start for Fedora, much less my ideal end states of it ending up in RHEL7/RHEL8.

[tmds]

I think I'm hitting the same issue, with a newer error message:

$ podman version
Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64
$ podman run --init busybox
Error: please specify a path to the container-init binary

[mheon]

Your libpod.conf probably has init_path = "" somewhere in it - try removing that line.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

We really need to get catatonit packaged in the Fedora/RHEL universe. @lsm5, @jnovy, do you have some spare cycles?

[vrothberg]

Friendly ping.

[oleastre]

As it seems easy to package I did it on copr.
https://copr.fedorainfracloud.org/coprs/oleastre/misc/package/catatonit/

However, I don't know what is the best to make it recognized by podman:

• install in /usr/bin and create a link to /usr/libexec/podman/catatonit
• always install to /usr/libexec/podman/catatonit, but this would make it only available for podman
• add a podman-catatonit package that depends on this one and creates the link
• update /usr/share/containers/libpod.conf to point to /usr/bin/catatonit

Here is a copy of my spec file, as simple as it can be.

Name:           catatonit
Version:        0.1.4
Release:        1%{?dist}
Summary:        A container init that is so simple it's effectively brain-dead.

License:        GPLv3+
URL:            https://github.com/openSUSE/catatonit
Source0:        https://github.com/openSUSE/catatonit/releases/download/v%{version}/catatonit.tar.xz

BuildRequires:  autoconf automake gcc glibc-static libtool

%description
A container init that is so simple it's effectively brain-dead. This is a
rewrite of initrs in C, because we found that it is not possible to statically
compile Rust binaries without using musl. That was, in turn, a reimplementation
of other container inits like tini and dumb-init.


%prep
%autosetup


%build
autoreconf -fi
%configure
%make_build


%install
rm -rf $RPM_BUILD_ROOT
%make_install


%files
%license COPYING
%doc README.md
%{_bindir}/catatonit


%changelog
* Tue Feb 18 2020 Olivier Samyn <[email protected]>
- Initial spec file based on catatonit README

[jamescassell]

Definitely /usr/bin IMO, seems podman should be able to check $PATH

[oleastre]

For info, I tried to change libpod.conf to poing init_path to /usr/bin/catatonit (after installing my package) but podman was still looking for /usr/libexec/podman/catatonit (default fedora 31 podman install)

[lsm5]

I have the old spec file at https://pagure.io/catatonit/blob/master/f/catatonit.spec . Could've sworn I had a package review created but can't find it on rhbz.

@oleastre are you interested in officially packaging it for Fedora?

[rhatdan]

@jnovy @lsm5 Any progress on getting catatonic packaged for Fedora?

[rhatdan]

If catatonic is not supposed to be run by a user then it should not go into /usr/bin. If it is not supposed to be run by an admin then it should not go in /usr/sbin. If it is only run by applications like container engines, then it should go into /usr/lib/catatonit/catatonit or /usr/libexec/catatonit/catatonit. or /usr/libeec/catatonit.

Then each distro could customize their install location in libpod.conf (containers.conf) going forward.

[oleastre]

@lsm5 I'm not a recognized fedora packager although I sometimes provide patches to existing one.
I just did it quickly to solve the problem here. But if this can solve the problem on the fedora level, why not.
And as suggested by @rhatdan , I'll update the spec file to install in /usr/libexec/catatonit/catatonit which makes more sense.
Although that does not solve the installation for podman on fedora. But I suppose this should be properly handled in the fedora bugzilla instead of here.

[lsm5]

@lsm5 I'm not a recognized fedora packager although I sometimes provide patches to existing one.
I just did it quickly to solve the problem here. But if this can solve the problem on the fedora level, why not.
And as suggested by @rhatdan , I'll update the spec file to install in /usr/libexec/catatonit/catatonit which makes more sense.
Although that does not solve the installation for podman on fedora. But I suppose this should be properly handled in the fedora bugzilla instead of here.

I'll post a fedora review request and @jnovy will review it. Patches are welcome anytime!

[lsm5]

package review request filed at https://bugzilla.redhat.com/show_bug.cgi?id=1804768

[ikke-t]

this is likely obvious to everyone, but also RHEL8 is missing it. Just to mention here in order to get it somehow into pipeline to RHEL.

podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64
podman run --init -d --name="home-assistant" -e "TZ=Europe/Helsinki" -v /home/redhatter/ha-config:/config -p 8123:8123 homeassistant/home-assistant:stable
Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory                                          

[lsm5]

this is likely obvious to everyone, but also RHEL8 is missing it. Just to mention here in order to get it somehow into pipeline to RHEL.

@ikke-t yes, this will quite likely happen once things are present in Fedora. But RHEL will have its own processes/schedules, so no ETA promised. But I can ensure the OBS packages for CentOS have it once Fedora is done.

[lsm5]

catatonit has been added to the OBS repos (debian and ubuntu only so far), will be published soon. I'll add it to CentOS once it's in Fedora.

[wwilson]

Who is currently maintaining catatonit? We reported a couple of serious bugs (encountered while using it with podman) over a month ago, but noticed that there hasn't been any active development in about a year. If the Project Atomic community is going to depend on it, would it make sense for us to take over maintenance as well?

[vrothberg]

@wwilson, catatonit is a _very_ small binary and not performing much work. A year without code changes doesn't necessarily imply the project is dead. Can you point to the bugs? Did you reach out to the maintainers?

[cyphar]

@wwilson Are these the bugs you're referring to -- openSUSE/catatonit#4 and openSUSE/catatonit#5? I didn't get the impression these were "serious bugs" in the reports -- they are semantic bugs and should be fixed but I haven't had time to look into them (I've been away from home almost consistently for various reasons since Jan 2nd). If those aren't the bugs you're referring to, then I don't believe I've heard from you.

I'm currently on vacation, so I don't have much time to write up patches (mainly because I'm not sure what the correct exit code semantics should be). But if you have a PoC patch or outline of what kind of fix would be appropriate I'd be happy to carry it.

[cyphar]

I've fixed the two bugs I linked, and released 0.1.5.

[vrothberg]

Thanks, @cyphar !

[cben]

I want to use --init in a script used on various distros, here's partial sample from googling (didn't actually test):

• [x] OpenSUSE: packaged catatonit, podman Requires it
• [x] Arch: packaged catatonit, optional dep of podman-docker, but not of podman itself?
• [ ] ubuntu via snap: didn't get it to work, but doesn't contain any file named catatonit
• didn't look into any of the kubic repos

For me --init is nice to have (more reliable killing), I can skip when missing. Can I assume most distros would put it at /usr/libexec/podman/catatonit?

Anyway for near future the portable solution seems to be to modify the image to contain tini or catatonit.

[cben]

No, can't assume a path. opensuse seems to put it at `/usr/bin, and libpod.conf points there.

[hakunin]

Is there a workaround right now? I'm new to Docker and I just can't get it to run on one of my machines and switching to Podman I'm stuck on this.

[mheon]

If you can get a tini or katatoinit binary installed on the host, the path to it is in libpod.conf - set the init_path variable to the path you have an init binary installed at. The default is "/usr/libexec/podman/catatonit" so you could also just install the binary to that location and not worry about changing config files.

[mheon]

Sorry, catatonit not katatoinit - been looking at Kata containers too much recently, I suppose

[lsm5]

Is there a workaround right now? I'm new to Docker and I just can't get it to run on one of my machines and switching to Podman I'm stuck on this.

If you're on ubuntu/debian you can get it from the kubic repos. See: https://software.opensuse.org//download.html?project=devel%3Akubic%3Alibcontainers%3Astable&package=catatonit

catatonit package is being reviewed for Fedora right now so it's not available in the base repos for Fedora or CentOS yet. If you don't mind installing an unreleased rpm directly to get unblocked right away, you can get it from https://lsm5.fedorapeople.org/catatonit-0.1.4-1.x86_64.rpm

[hakunin]

Thank you SO MUCH! The package works perfectly.

[jpeeler]

@lsm5 this looks like it is stalled with a needinfo for you?
https://bugzilla.redhat.com/show_bug.cgi?id=1804768

[lsm5]

@lsm5 this looks like it is stalled with a needinfo for you?
https://bugzilla.redhat.com/show_bug.cgi?id=1804768

yup, sorry, distractions. I'll get that going tomorrow..

[lsm5]

catatonit is now in the fedora testing repos https://bodhi.fedoraproject.org/updates/?packages=catatonit . I'd recommend people install this instead of any prior builds I posted.

It's also present on the OBS repos now except for CentOS Stream which has some dependency issues on the rpm package which can only be fixed by OBS admins.

[rhatdan]

Should probably add a recommends on suggests for this package to podman.

[lsm5]

Should probably add a recommends on suggests for this package to podman.

It's in there already for podman 1.9.1

[rhatdan]

Awesome.

[lsm5]

Fedora users, please add karma to the f31 and f32 packages https://bodhi.fedoraproject.org/updates/?packages=catatonit