Podman: ERRO[0057] unable to write pod event: "write unixgram @00017->/run/systemd/journal/socket: sendmsg: no such file or directory"

[chenjianxin]

OS: WSL 2 ubuntu

podman run --rm --volume="$PWD:/srv/jekyll" -p 127.0.0.1:4000:4000 -it jekyll/jekyll:pages jekyll serve

Trying to pull docker.io/jekyll/jekyll:pages...
Getting image source signatures
Copying blob 278f4c997324 done
Copying blob c69cba5b7867 done
Copying blob 9d48c3bd43c5 done
Copying blob 9ce9598067e7 done
Copying blob 867dd521f6d0 done
Copying blob 49e3e26660b5 done
Copying config a572804619 done
Writing manifest to image destination
Storing signatures

ERRO[0057] unable to write pod event: "write unixgram @00017->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0059] unable to write pod event: "write unixgram @00017->/run/systemd/journal/socket: sendmsg: no such file or directory"
Error: slirp4netns failed

[mheon]

You are seeing two separate issues here.

The write unixgram errors are attempts to write to the systemd journal, which doesn't seem to be present on WSL images.

The slirp4netns issue was the fatal issue.

[rhatdan]

That is a pretty poor error message on slirp4netns. Is this just the package is missing or it failed and did not give us a decent message.
@AkihiroSuda FYI

[rhatdan]

Should the messages be logrus.Warning if the systemd is not running, the only issue here is that if the user looked for events he would not see them correct?

[rhatdan]

@chenjianxin Could you try the same command and see if it works if you do
--net=host
Which should eliminate slirp4netns, I believe.

[AkihiroSuda]

@rhatdan slirp4netns should be printing some errors, but it seems podman ignores that

[rhatdan]

https://github.com/containers/libpod/pull/4338 should help get those messages.

[chenjianxin]

4338 should help get those messages.

podman run -d --name mysql-server --net=host -e MYSQL_ROOT_PASSWORD=secret mysql:latest

ERRO[0001] unable to write volume event: "write unixgram @00029->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0001] unable to write pod event: "write unixgram @00029->/run/systemd/journal/socket: sendmsg: no such file or directory"
Error: container_linux.go:346: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"mountpoint for cgroup not found\"": OCI runtime error

sudo podman network create -d bridge my-network
ERRO[0000] unable to write system event: "write unixgram @0000d->/run/systemd/journal/socket: sendmsg: no such file or directory"
/etc/cni/net.d/my-network.conflist

[rhatdan]

What is /etc/cni/net.d/my-network.conflist and why is the container trying to use it?

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[vrothberg]

What is /etc/cni/net.d/my-network.conflist and why is the container trying to use it?

@chenjianxin, are you still seeing the issue?

[rdcoe]

Should this issue remain closed? Still seeing it:

$ cat /etc/debian_version
buster/sid
$ uname -a
Linux BEASTIE 4.19.84-microsoft-standard #1 SMP Wed Nov 13 11:44:37 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ podman run --net=host -it fedora bash
ERRO[0000] unable to write pod event: "write unixgram @0000d->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0000] unable to write pod event: "write unixgram @0000d->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0000] unable to write pod event: "write unixgram @0000d->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0000] unable to write pod event: "write unixgram @0000d->/run/systemd/journal/socket: sendmsg: no such file or directory"
# exit
ERRO[0330] unable to write pod event: "write unixgram @0000d->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0331] unable to close namespace: "close /proc/119/ns/user: bad file descriptor"

[vrothberg]

Thanks for jumping in @rdcoe! Maybe you can answer @rhatdan 's question:

What is /etc/cni/net.d/my-network.conflist and why is the container trying to use it?

[vrothberg]

Actually, your output looks different. Can you do a podman info? There seems to be no systemd in WSL2. Does a podman --cgroup-manager=cgroupfs run ... work?

[rdcoe]

$ podman --cgroup-manager=cgroupfs run --net=host -it fedora bash
ERRO[0000] unable to write pod event: "write unixgram @0001b->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0000] unable to write pod event: "write unixgram @0001b->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0000] unable to write pod event: "write unixgram @0001b->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0000] unable to write pod event: "write unixgram @0001b->/run/systemd/journal/socket: sendmsg: no such file or directory"

[rhatdan]

How about --events-backend=file

The default for windows should be set to file not to journal

[rdcoe]

--events-backend=file fixed it. Thanks!

Can this be made the default through config, along with the other relevant flags?

[rhatdan]

We are working on containers.conf right now, to allow different distros and users to modify the defaults.

https://github.com/containers/common

[rdcoe]

Cool. So should this issue be closed now?

[rhatdan]

@rdcoe Is everything else working on WSL Now?

[rdcoe]

Too new to podman to know if "everything" is working but this particular issue is solved.

[mheon]

No need to wait for containers/common, events_logger = "file" in libpod.conf should set things properly

[rdcoe]

Even better. That config also fixed the error I was seeing on exiting the container:

before the config change:

$ podman --cgroup-manager=cgroupfs --events-backend=file run --net=host -it fedora bash
[root@BEASTIE /]# exit
ERRO[0988] unable to close namespace: "close /proc/119/ns/user: bad file descriptor"

after setting the libpod.conf to use a file instead of journald

$ podman run --net=host -it fedora bash
[root@BEASTIE /]# exit
$

[mathieu-benoit]

Hi there, I'm very new with podman and I'm trying to get it working with WSL1 but I still have the error with the command below:

$ podman --cgroup-manager=cgroupfs --events-backend=file run --net=host -it fedora bash
Error: container_linux.go:346: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"EOF\"": OCI runtime error

Are you able to have this command working? I see otherwise that "setting the libpod.conf to use a file instead of journald" seems to work for you @rdcoe and @mheon but sorry for a dummy question, but how to do that?
For my understanding is it going to be fixed with podman version 1.6.4?
Thanks!
NB: I'm currently with podman version 1.6.2 (currently available here) | Ubuntu 18.04 on WSL1.

[rhatdan]

My understanding is podman did not work well on WSL1 because of the way the kernel was configured. Is there a chance you can try with WSL2?

[mathieu-benoit]

Thanks @rhatdan, I just converted my WSL1 into WSL2 and indeed this command below works on WSL2:
podman --cgroup-manager=cgroupfs --events-backend=file run --net=host -it fedora bash. Thanks!

[PavelSosin-320]

@rhatdan Hi I'm working on WSL! I get this message! This is the known gap - WSL doesn't support systemd because it uses MS proprietary init. On the other hand, Ubuntu 20.04 from Microsoft store does support systemd as I can see in its startup message. So I think it is possible for all distros but not documented. Ubuntu 20.04 is the only distro in which systemd is the root process.

[mheon]

The net=host bit is concerning to me and should be looked into - I'd hope that we'd be able to get standard networking running, given WSL2's support for Docker (and presumably Libnetwork).

The remaining two (--cgroup-manager=cgroupfs and --events=backend=file) are not a big deal - they're the usual defaults when systemd is not available, and will be safe even the systemd does use systemd for PID1. We may want to investigate improving our defaults here - if systemd is not available we should automatically be selecting the safe option.

[PavelSosin-320]

There is additional complexity in the Podman networking: Pod creation. I'm building Pod which consists of 2 images:
podman pod create --name theiaTest -p 3000:3000
podman run -dt --pod theiaTest -it -P -v "$(pwd):/home/project:cached" --name=theia theiaide/theia
podman run -dt --pod theiaTest --name=ngInx
and the main issue is
Error: error starting some containers: internal libpod error due to
les-legacy tables present, use iptables-legacy to see them
iptables v1.8.4 (nf_tables): CHAIN_ADD failed (No such file or directory): chain PREROUTING
ERRO[0000] Error while adding pod to CNI network "podman": unable to create chain CNI-HOSTPORT-SETMARK: running [/usr/sbin/iptables -t nat -S --wait]: exit status 4: # Warning: iptables-legacy tables present, use iptables-legacy to see them
iptables v1.8.4 (nf_tables): CHAIN_ADD failed (No such file or directory): chain PREROUTING
CentOS8 has got nft. I removed (renamed) iptables configuration to designate that iptables are not directly manageable. It will be nice if libpod will check firewall capability and simply warn/skip firewall management if Podman runs inside VM container.
P.S. In my VM I see both eth0 and cni-podman0. eth0 is the interface to the host's network and it is known only after VM starts.
WSL generates /etc/hosts file for docker which contains Host machine DNS names. All services running in WSL VM must listen 0.0.0.0.0 , i.e. both localhost and eth0.

[PavelSosin-320]

Ubuntu 20.04 from Microsoft store has Podman from the Kubic project ??????

• Documentation: https://help.ubuntu.com
  
  
  
  • Management: https://landscape.canonical.com
  
  
  • Support: https://ubuntu.com/advantage
  
  

System information as of Tue Sep 8 16:12:38 IDT 2020

System load: 1.43 Processes: 73
Usage of /: 2.4% of 250.98GB Users logged in: 0
Memory usage: 9% IPv4 address for cni-podman0: 10.88.0.1
Swap usage: 0% IPv4 address for eth0: 172.28.207.181

• Kubernetes 1.19 is out! Get it in one command with:

  sudo snap install microk8s --channel=1.19 --classic

  https://microk8s.io/ has docs and details.
  And starts from systemd? It doesn't work but they are in the right direction.

[georgettica]

Just to raise the issue.
Still on wsl2 there is a need for the flags

I am running Debian inside my win10.

podman run --events-backend=file --net=host --rm -it python bash