Podman: No support for journald logging

[Shulito]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I'm running Ubuntu 18.04 and today I updated my packages (podman included) and now I'm getting this error when trying to run any podman command:

Error: could not get runtime: eventer creation: No support for journald logging

Steps to reproduce the issue:

1. Update packages on Ubuntu 18.04.
2. Run podman ps.

Describe the results you received:
Error on the console.

Describe the results you expected:
The list of containers.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.3.1-dev
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

Error: could not get runtime: eventer creation: No support for journald logging

Additional environment details (AWS, VirtualBox, physical, etc.):

[countzero1942]

I get the same error on a remote server running Ubuntu 18.04.2 LTS ("minimum install") with "podman info". Followed site installation instructions. (Except I used 'apt' instead of 'apt-get'.) The journald service appears to be running properly: 'active' status.

[mheon]

3128 should fix

[countzero1942]

As far as I can tell, "podman version 1.3.1-dev" is toxic. (From "podman --version".)

Yesterday I installed Ubuntu 18.04 LTS on a new remote server. Installed podman according to site instructions. Then "podman info" (first command you run) crashes with the above error.

So I went on my ad-hoc laptop home-server running only Ubuntu 18.04 and typed "podman --version". It was "1.3.0". It ran fine. So I uninstalled podman and reinstalled. Now it's "1.3.1-dev". And "podman info" crashes with the same error.

The remote server is Hetnzer which uses its own Ubuntu repository mirror. The laptop uses the default Ubuntu repository.

So it would seem that going back to 1.3.0 would be one way of fixing it. (Not exactly sure how to do that yet. The documentation for this is not yet implemented on the podman.io site.)

Update: the PPA repository doesn't presently support choosing older builds or tags like "latest" and "nightly".

[chpio]

Why is the official ubuntu ppa (https://launchpad.net/~projectatomic/+archive/ubuntu/ppa) releasing broken dev builds?

So it would seem that going back to 1.3.0 would be one way of fixing it. (Not exactly sure how to do that yet. The documentation for this is not yet implemented on the podman.io site.)

I think, you can not do that, at least not on ubuntu.

[chpio]

3128 should fix

is there a workaround for that fix, that would work right now?

[isavcic]

@chpio

I think, you can not do that, at least not on ubuntu.

You can, only if they've left the older versions up on the PPA.

[isavcic]

Ok, just opened #3129, because I don't see a reason not to provide stable (release) packages alongside dev ones for Ubuntu.

[isavcic]

@chpio

3128 should fix

is there a workaround for that fix, that would work right now?

https://github.com/containers/libpod/pull/3128#issuecomment-492647345

[mheon]

@lsm5 What build tags are we shipping for ubuntu? It looks like this broke over there because the PPA builds don't have the new systemd tag.

[undecaf]

@countzero1942 @chpio @isavcic How to to reinstall Ubuntu Podman v1.3.0 for the time being

View package details on the Project Atomic PPA page, filter by "podman" and "superseded" and expand entry podman - 1.3.0-1~dev~ubuntu18.04~ppa20.

Download and install podman_1.3.0-1~dev~ubuntu18.04~ppa20_amd64.deb

[rhatdan]

Yes we are working on snap releases which should be the stable releases, and we will continue to provide latest releases.

[baude]

the simplest thing to do here is rebuild the current package but make sure to dep libsystemd-dev fwiw.

[mheon]

Agree - change build tags and libsystemd-dev as a req

[mheon]

Also - how are we building the Ubuntu packages for the PPA?

Can we automate that as part of PR testing?

[isavcic]

@rhatdan

Yes we are working on snap releases which should be the stable releases, and we will continue to provide latest releases.

What is the logic behind the decision to provide dev packages, but stable snap? Is there a reason not to do the deb for stable as well? Snap isn't used on servers, at least not often.

[mheon]

From what I understand, packaging difficulty - we don't have anyone experienced with .deb packaging, so our packager is trying to produce snaps as an easier way to get Podman packaged for non-RPM distros.

We also have efforts ongoing to get Podman packaged in upstream Debian, but the pace there is very slow, and it could be a long time before we have success there.

If anyone wants to volunteer to assist in packaging for Ubuntu, that would help a lot.

[isavcic]

@mheon but you're already making dev deb packages for the PPA?

[mheon]

Auto-building off the latest master - it was an effort to set up initially, but lower overall maintenance as the builds themselves don't need to be touched (until things like this happen, master breaks, and the builds no longer work). It's a lot less of a time/resource investment than tracking the packages in the Projectatomic PPA for new releases and manually building them.

[alitvak69]

So a fix was applied to cri-o-runc regression tonight by wonderful developers cooperating with each other, can the podman 1.3.1 be fixed as well? Thank you all for your hard work and dedication.

[mheon]

I was hoping to have it out yesterday, but we're presently experiencing a number of CI issues. I'm hopeful that we can get it out later today, though.

If you're interested in tracking the release process, remaining PRs before we can cut a release are at https://github.com/containers/libpod/pulls?q=is%3Aopen+is%3Apr+label%3A%22Release+Notes+1.3.1%22

[alitvak69]

Thank you again.

[baude]

a fix has been merged

[alitvak69]

As I understand all hurdles were cleared yesterday. Can you please push a new build to ppa?

[mheon]

@lsm5 We're auto-building master, right? Shouldn't it have picked up the latest commits by now?

[jpic]

3128 works for me on ubuntu, note that i had to rm ~/.config/containers.

[mheon]

Eeek. That shouldn't have been necessary.

[jpic]

It was for me:

newfstatat(AT_FDCWD, "/home/jpic/.local/bin/fuse-overlayfs", 0xc4208005e8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/jpic/.local/bin/fuse-overlayfs", 0xc4208006b8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/jpic/.config/containers/storage.conf", {st_mode=S_IFREG|0664, st_size=708, ...}, 0) = 0
openat(AT_FDCWD, "/home/jpic/.config/containers/storage.conf", O_RDONLY|O_CLOEXEC) = 5
epoll_ctl(4, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3041177344, u64=139838586400512}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 5, 0xc42082399c) = -1 EPERM (Operation not permitted)
fstat(5, {st_mode=S_IFREG|0664, st_size=708, ...}) = 0
read(5, "[storage]\n  driver = \"vfs\"\n  run"..., 1220) = 708
read(5, "", 512)                        = 0
close(5)                                = 0
pwrite64(3, "\3\0\0\0\0\0\0\0\20\0\2\0\0\0\0\0\4\0\0\0\0\0\0\0\5\0\0\0\0\0\0\0"..., 4096, 12288) = 4096
fdatasync(3)                            = 0
pwrite64(3, "\1\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\355\332\f\355\2\0\0\0\0\20\0\0\0\0\0\0"..., 4096, 4096) = 4096
fdatasync(3)                            = 0
munmap(0x7f2eb5504000, 32768)           = 0
flock(3, LOCK_UN)                       = 0
close(3)                                = 0
geteuid()                               = 1000
write(2, "Error: could not get runtime: ev"..., 80Error: could not get runtime: eventer creation: No support for journald logging
) = 80
newfstatat(AT_FDCWD, "/etc/containers/registries.conf", {st_mode=S_IFREG|0644, st_size=2978, ...}, 0) = 0
exit_group(125)                         = ?
+++ exited with 125 +++
jpic@ci:~/buildah/src/github.com/containers/libpod$ cat /home/jpic/.config/containers/storage.conf
                                                                                                                                                                                                                                                                            [190/1931]
jpic@ci:~/buildah/src/github.com/containers/libpod$ cat /home/jpic/.config/containers/storage.conf
[storage]
  driver = "vfs"
  runroot = "/tmp/1000"
  graphroot = "/home/jpic/.local/share/containers/storage"
  [storage.options]
    size = ""
    remap-uids = ""
    remap-gids = ""
    remap-user = ""
    remap-group = ""
    ostree_repo = ""
    skip_mount_home = ""
    mount_program = ""
    mountopt = ""
    [storage.options.thinpool]
      autoextend_percent = ""
      autoextend_threshold = ""
      basesize = ""
      blocksize = ""
      directlvm_device = ""
      directlvm_device_force = ""
      fs = ""
      log_level = ""
      min_free_space = ""
      mkfsarg = ""
      mountopt = ""
      use_deferred_deletion = ""
      use_deferred_removal = ""
      xfs_nospace_max_retries = ""
jpic@ci:~/buildah/src/github.com/containers/libpod$ ls /home/jpic/.config/containers/
libpod.conf   storage.conf
jpic@ci:~/buildah/src/github.com/containers/libpod$ cat /home/jpic/.config/containers/libpod.conf
volume_path = "/home/jpic/.local/share/containers/storage/volumes"
image_default_transport = "docker://"
runtime = "runc"
conmon_path = ["/usr/libexec/podman/conmon", "/usr/libexec/crio/conmon", "/usr/local/lib/podman/conmon", "/usr/local/libexec/crio/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/lib/podman/bin/conmon", "/usr/lib/crio/bin/conmon"]
conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
cgroup_manager = "cgroupfs"
init_path = "/usr/libexec/podman/catatonit"
static_dir = "/home/jpic/.local/share/containers/storage/libpod"
tmp_dir = "/run/user/1000/libpod/tmp"
max_log_size = -1
no_pivot_root = false
cni_config_dir = "/etc/cni/net.d/"
cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"]
infra_image = "k8s.gcr.io/pause:3.1"
infra_command = "/pause"
enable_port_reservation = true
label = true
network_cmd_path = ""
num_locks = 2048
events_logger = "journald"
EventsLogFilePath = ""

[runtimes]
  runc = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc"]
jpic@ci:~/buildah/src/github.com/containers/libpod$ rm -rf /home/jpic/.config/containers/
jpic@ci:~/buildah/src/github.com/containers/libpod$ bin/podman
Error: missing command 'podman COMMAND'
Try 'podman --help' for more information.
#### ^^^^ GOOD -> that means podman passes the error

[jpic]

Diff between old and working configs is just the events_logger indeed:

jpic@ci:~/.config/containers$ diff -u libpod.old.conf libpod.conf 
--- libpod.old.conf     2019-05-16 16:42:03.254969756 +0200
+++ libpod.conf 2019-05-16 16:25:56.495492027 +0200
@@ -17,7 +17,7 @@
 label = true
 network_cmd_path = ""
 num_locks = 2048
-events_logger = "journald"
+events_logger = "file"
 EventsLogFilePath = ""

 [runtimes]
jpic@ci:~/.config/containers$ diff -u storage.old.conf storage.conf 
jpic@ci:~/.config/containers$

[alitvak69]

Were you using a self built package? I don't see an update in ppa

[jpic]

@alitvak69 yes, but try changing events_logger from "journald" to "file" maybe that'll do.

If i change events_logger from "file" to "journald" again: Error: could not get runtime: eventer creation: No support for journald logging

[alitvak69]

That was the point of fixing so one doesn't need to change the default config. Isn't it right?

[mheon]

Assuming the config is included in the package, it should be fine...

It might not be so for rootless users, though.

I'll poke @lsm5 to get the systemd build tag included as well, so the 'journald' option works as well

[alitvak69]

The other question, and forgive me for being a pest, why not journald? I thought podman supports both file and journal.

[alitvak69]

Thank you

[mheon]

Journald is gated behind a build tag as of 1.3.1 and we haven't included the tag in the PPA builds yet - we'll add it for future builds so it's enabled.

[alitvak69]

Can I get all of this via snap now? If yes, can you please post some quick instructions?

[jpic]

@alitvak69 didn't changing events_logger from "journald" to "file" workaround your blocker ?

[mheon]

I've talked with @lsm5 and future builds should include journald support by default.

[alitvak69]

@jpic It is not a very good one as it means changing / adding config in many places in my environment.
@mheon I appreciate your help and sorry for being impatient. I just thought that since you doing snaps, I could get the latest 1.3.1 faster.

[mheon]

I don't know if those (snaps) are ready yet - I'll ask

[jpic]

@alitvak69 ok good luck then, because events_logger=journald still makes podman exit with No support for journald logging here on master build.

[alitvak69]

@jpic Understand, but currently I don't even have access to that unless I build it myself.

[alitvak69]

New ppa build failed so far.

[jpic]

@alitvak69 Actually building master with BUILDTAGS="systemd" does make podman with events_logger="journald" work here on ubuntu 18.04. Sorry for the misguidance, hope to improve with time.

[countzero1942]

@countzero1942 @chpio @isavcic How to to reinstall Ubuntu Podman v1.3.0 for the time being

View package details on the Project Atomic PPA page, filter by "podman" and "superseded" and expand entry podman - 1.3.0-1~dev~ubuntu18.04~ppa20.

Download and install podman_1.3.0-1~dev~ubuntu18.04~ppa20_amd64.deb

This worked for me.

wget "https://launchpad.net/~projectatomic/+archive/ubuntu/ppa/+files/podman_1.3.0-1~dev~ubuntu18.04~ppa20_amd64.deb"

sudo apt install ./podman_1.3.0-1~dev~ubuntu18.04~ppa20_amd64.deb

Update:

To prevent podman 1.3.1-dev from reinstalling with "sudo apt update && sudo apt upgrade" you have to comment out the ppa repository inside:

/etc/apt/sources.list.d/projectatomic-ubuntu-ppa-bionic.list

[alitvak69]

@jpic You are absolutely correct, it works, as I built myself and tried. But I would definitely want a package built to succeed. May an off-topic question. I am using journal for podman events however conmon has is logging option -l /var/lib/containers/storage/overlay-containers/3ea0e5faae0b3c95f741e97ac764381f70e4ce98ff2284ecd2482856b2a43a52/userdata/ctr.log. Why do I have to have this file? It actually gets big if a container is in use and no standard tools to control rotation.

[haircommander]

@alitvak69 hey, I'm the conmon journald logging person. Adding journald logging to podman is on my list of things to do. I hope to have that out by podman 1.4 :)

[alitvak69]

@haircommander It is good to know you are working on it.

[alitvak69]

I don't want to bother anyone but there is no way to message ppa maintainer. Build failed, but as far as I know master build with the same tags works. Can the package build be fixed and retried?

[alitvak69]

`
/usr/bin/go build -ldflags '-X main.gitCommit= -X main.buildInfo=1558107955' -tags "seccomp systemd containers_image_ostree_stub" -o bin/podman github.com/containers/libpod/cmd/podman
go: disabling cache (/sbuild-nonexistent/.cache/go-build) due to initialization failure: mkdir /sbuild-nonexistent: permission denied

install: cannot stat 'bin/podman-remote': No such file or directory
Makefile:265: recipe for target 'install.bin' failed
make[2]: * [install.bin] Error 1
make[2]: Leaving directory '/<>'
debian/rules:54: recipe for target 'override_dh_auto_install' failed
make[1]: [override_dh_auto_install] Error 2
make[1]: Leaving directory '/<>'
debian/rules:33: recipe for target 'binary' failed
make: ** [binary] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2
`

[rhatdan]

@lsm5 @jnovy PTAL

[rufoa]

Build of 1.3.2-1~dev source package from PPA fails for me due to missing bin/podman-remote, similar to above

Build works when this patch is applied: https://github.com/containers/libpod/pull/3152/files
and journald logging problem is then resolved

So I think the solution here is to just merge that patch

[alitvak69]

This is rpm build in your case. I wish someone shared a deb build files.
I have little experience generating debs from source.

[rufoa]

This is rpm build in your case

Why do you say that? No - I am referring to the failing deb builds on the PPA.

When rhatdan's patch, linked above, is applied to the latest source package on launchpad, podman successfully compiles and installs with:

dpkg-buildpackage --no-sign -rfakeroot -b
sudo dpkg -i ../podman_1.3.2-1~dev~ubuntu18.04~ppa2_amd64.deb

and the "No support for journald logging" error is resolved.

This is what needs to be done to fix the PPA build failure which is happening.

[alitvak69]

I am sorry, my mistake. As I said little to non experience on building debs.

[alitvak69]

Hopefully @rufoa suggestion can be heard by ppa maintainers.

[alitvak69]

Build is still failing. I will attempt to use @rufoa suggestions and do a manual deb build for now. This is what I wanted to avoid in our automation scripts but oh well ... I hope someone here will be able to build a package eventually. I also hope that ppa may have either its own issue page or a mailing list.

[alitvak69]

The build is successful, thank you for your help @rufoa. I installed it and the journald problem is resolved via PPA package installation now.

[rufoa]

Resolved and can be closed. Thanks

[mheon]

Glad to hear it. If anyone is still getting failing builds off the PPA, feel free to comment and I'll reopen.

[karl-tpio]

Sorry to those subscribed on this thread for the notification on a closed thread...

Can i get some clarification on what's needed to get podman working with journald logging? As far as i understand this thread, podman after 1.3.2 should work with journald, correct? But this appears to be only when built manually and not for the packages in the PPA?

Given the information below, is there something i'm missing?

---

Attempt to run a container with journald:

root@my-podman-host:/etc/cni/net.d# podman run --log-driver=journald hello-world
[conmon:e] Include journald in compilation path to log to systemd journal

root@my-podman-host:/etc/cni/net.d# podman version
Version:            1.5.0
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

my environment details:

root@my-podman-host:/etc/apt/sources.list.d# cat projectatomic-ubuntu-ppa-bionic.list
deb http://ppa.launchpad.net/projectatomic/ppa/ubuntu bionic main
# deb-src http://ppa.launchpad.net/projectatomic/ppa/ubuntu bionic main

root@my-podman-host:/etc/apt/sources.list.d# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"

The exact install steps:

apt-get install -qq -y software-properties-common uidmap
add-apt-repository -y ppa:projectatomic/ppa
apt-get update -qq
apt-get -qq -y install podman

[haircommander]

@karl-tpio ah, I see we don't ship with journald support. You can download the libsystemd-journal package from apt-get, then build conmon from source https://github.com/containers/conmon and install it where needed (make podman PREFIX=/usr)

[mheon]

Do we need changes from @lsm5 to get conmon building with the right dependencies?

[haircommander]

we would, yes

[haircommander]

if we want to ship with journald by default

[karl-tpio]

@haircommander I'd be a _very_ strong thumbs up for shipping with journald support by default. With each new Ubuntu version, more things are being transitioned to it. I don't expect plain-text logging to go away in my lifetime, but i do expect JSON formatted logs to grow obsolete in favor of the just as capable binary format that journald uses in the next few years as people update their log collectors to support the far more compact/easier-to-parse binary format behind journald.

Should i open a new issue on this repo to get the ball rolling? I am not a particularly competent build engineer, but i may be able to get in a PR if i can get a bit of guidance as to _where_ the build automation/configs are located.

[haircommander]

yes if you could open an issue that'd be great @karl-tpio! I'll then assign our main packager and we can all collab on a solution there

[alitvak69]

I am a bit confused. I believe that the issue is with conmon and not podman. Podman deb supports journald already. At least it was when this ticket was closed, but conmon continued writing into files. I was told by @haircommander he was working on the journald support then. So if podman is built with the systemd support already, shouldn't we take care of conmon journald instead and reinstall conmon after rebuilding it?

[karl-tpio]

@alitvak69 podman is the user facing program that coordinates all the little bit that must go together to get a container running. There's next to nothing of value that that podman would output as log stream (of course, this assumes you're not actively debugging/developing podman).

AFAIK, the users choice to use journald is nothing more than a flag that podman passes down to the actual process that forks off and then becomes the container.

@haircommander I'll get that open shortly. To be clear, that ticket should be on the common repo, yes?

[alitvak69]

@karl-tpio Not disputing anything you said, I just wanted to make sure which piece needs to be rebuilt. And since conmon had previously no journald support, I wondered if rebuilding it with journald support would be enough.

[alitvak69]

One more thing, there is no libsystemd-journal. It would be libsystemd-dev most likely.