toolbox broken w/o sssd-kcm and /var mounted nodev,noexec,nosuid

[rusty-snake]

I first tried toolbox with fedora 31 at the begin of this year, but this was juts broken. I assumed it was because messed up this installation. I now reinstalled fedora 32 and tried toolbox again, but it was still broken. It just shows me one of the following error messages:

• toolbox: failed to start container fedora-toolbox-32
• toolbox: invalid entry point PID of container toolbox

Since it was a new installation there was no long history of system change, but I still
needed 6 test VMs and 3 days to find the issues:

TL;DR: toolbox is broken with

1. sss-kcm.socket disabled (sudo disable sssd-kcm.socket, to test: also disable sssd.service)
2. /var mounted with at least one of nodev,noexec,nosuid.

Now I have two questions:

1. Why does it need sssd-kcm and /var: dev,exec,suid (i want to understand it)
2. Is there any way that toolbox works w/o sssd-kcm or /var mounted nodev,noexec,nosuid

If 2 is not possible you should consider printing better error messages instead of failed to start (why does it fail?).

Again, to be clear so there are no misunderstandings. sssd-kcm and /var are two independent thing which break toolbox.

[debarshiray]

sssd-kcm.socket and the mount options of /var on the host are two different things.

The former is necessary to access the host's Kerberos credentials cache from inside the container, to make it easier to use Kerberos.

Toolbox uses podman create --volume /var:/run/host/var:rslave ... and it sounds like this doesn't work with the way /var is mounted on your system. The rslave flag is used to ensure that new mount points that appear under the host's /var also show up inside the container under /run/host/var. I am curious why you have different mount flags for /var - what's the use-case?

[debarshiray]

I first tried toolbox with fedora 31 at the begin of this year, but this was juts broken.
I assumed it was because messed up this installation. I now reinstalled fedora 32
and tried toolbox again, but it was still broken. It just shows me one of the following
error messages:

* `toolbox: failed to start container fedora-toolbox-32`

* `toolbox: invalid entry point PID of container toolbox`

I suspect these were regressions in Podman.

[debarshiray]

It seems that you are using the older POSIX shell implementation of Toolbox, because the new Go implementation got better at (softly) handling the sssd-kcm.socket situation as it was always meant to be.

Once you sudo systemctl disable sssd-kcm.socket, and reboot, the following command shows a socket file that doesn't actually exist on the file system:

$ systemctl show --value --property Listen sssd-kcm.socket

The POSIX shell implementation didn't check if this file actually exists or not, and would blindly try to bind mount it while creating the container. The Go implementation does the check. If the socket is missing, it skips trying to bind mount it, and the container gets created without the Kerberos integration.

[rusty-snake]

The former is necessary to access the host's Kerberos credentials cache from inside the container, to make it easier to use Kerberos.

It seems that you are using the older POSIX shell implementation of Toolbox, because the new Go implementation got better at (softly) handling the sssd-kcm.socket situation as it was always meant to be.

The POSIX shell implementation didn't check if this file actually exists or not, and would blindly try to bind mount it while creating the container. The Go implementation does the check. If the socket is missing, it skips trying to bind mount it, and the container gets created without the Kerberos integration.

$ toolbox --version
toolbox version 0.0.93
$ systemctl is-active sssd-kcm.socket 
inactive
$ toolbox enter        
Error: failed to start container fedora-toolbox-32
$ sudo systemctl start sssd-kcm.socket
$ toolbox enter
-- works --
$ sudo systemctl stop sssd-kcm.socket
$ toolbox enter
-- works --

I need to do some more tests.

I am curious why you have different mount flags for /var - what's the use-case?

Security, see https://wiki.archlinux.org/index.php/Security#Mount_options for example. However I moved away for they, since they seem to also break systemd-nspawn.

[debarshiray]

$ toolbox --version
toolbox version 0.0.93
$ systemctl is-active sssd-kcm.socket 
inactive
$ toolbox enter        
Error: failed to start container fedora-toolbox-32
$ sudo systemctl start sssd-kcm.socket
$ toolbox enter
-- works --
$ sudo systemctl stop sssd-kcm.socket
$ toolbox enter
-- works --

I need to do some more tests.

Let's dig deeper. What happens if you do this after attempting the failed enter:

$ podman stop fedora-toolbox-32
$ podman start --attach fedora-toolbox-32
...

[rusty-snake]

Error: unable to start container XXX: error stat'ing file `/run/.heim_org.h5l.kcm-socket`: No such file or directory: OCI runtime command not found error
zsh: exit 125   podman start --attach fedora-toolbox-32

$ systemctl cat sssd-kcm.socket
# /usr/lib/systemd/system/sssd-kcm.socket
[Unit]
Description=SSSD Kerberos Cache Manager responder socket
Documentation=man:sssd-kcm(8)


[Socket]
ListenStream=/run/.heim_org.h5l.kcm-socket

[Install]
WantedBy=sockets.target

[rusty-snake]

I just created a new toolbox-container with

$ toolbox rm fedora-toolbox-32
$ toolbox create
Created container: fedora-toolbox-32
Enter with: toolbox enter
$ toolbox enter
-- works

and it works. Maybe the issue with sssd-kcm is gone for new toolbox-containers, will test again after the next restart.

---

Regarding the /var mount option: I don't think there is much what toolbox can do expect good/helpfull error messages (IDK what the current is).

[debarshiray]

The thing is that the KCM socket at /run/.heim_org.h5l.kcm-socket is bind mounted into the container when it's initially created. If you created the container when the socket was present, and then later removed the socket, then the container will fail to start.

[debarshiray]

I am curious why you have different mount flags for /var - what's the use-case?

Security, see https://wiki.archlinux.org/index.php/Security#Mount_options
for example. However I moved away for they, since they seem to also break
systemd-nspawn.

How does it break systemd-nspawn?

I don't know why a /var that's mounted with nodev,noexec,nosuid on the host can't be bind mounted as --volume /var:/run/host/var:rslace, but if both Podman and systemd-nspawn are having problems then it seems like there's some valid reason.

[debarshiray]

It seems to me that the only actionable thing here is to make the containers more resilient to the KCM socket coming and going. Since the host's /run is also available inside the container at /run/host/run, it's possible to bind mount the socket when the container starts up, as opposed to when it's created.

However, the problem is that the location of the socket can be changed. Currently we extract it from the sssd-kcm.socket unit on the host via systemd when the container is created.

I guess we could extract the path to the socket from /run/host/usr/lib/systemd/system/sssd-kcm.socket inside the container instead of asking systemd via D-Bus on the host. However, without having a clear use-case, I don't know if it would be worth it.

Is KCM not working well for you?

[rusty-snake]

How does it break systemd-nspawn?

If the root is in /var/lib/machines (the default) machinectl shell fails. I don't remember the exact error but it was something about pty.

Is KCM not working well for you?

I not use/need sssd/kerberos, that's why I disabled it.

[debarshiray]

Is KCM not working well for you?

I not use/need sssd/kerberos, that's why I disabled it.

Ok. Sounds like you are not toggling KCM on and off all the time, so the current status quo doesn't seem so bad.

[debarshiray]

I am tempted to close this.

The code now skips KCM configuration if it's disabled when creating the container, and without a pressing use-case I am hesitant to add a bunch of complications to enable toggling KCM on and off during the lifetime of a container.