Runc: [rootless+cgroup2+CRI] runc did not terminate successfully: lstat /sys/fs/cgroup/user.slice/user-1001.slice/[email protected]/user.slice/cri-containerd-<id>.scope: no such file or directory\n: unknown"

[AkihiroSuda]

crictl stop fails on rootless + cgroup2 (systemd) + containerd + runc @ 0f7ffbebeb9e3e697dd0efc6f4ab9dc14054960e . The issue does not happen on crun v0.13.

Host

Ubuntu 20.04, systemd 245 (245.4-4ubuntu3.1), kernel 5.4.0-33-generic #37-Ubuntu

containerd log

ERRO[2020-06-01T17:37:22.471213048+09:00] Failed to handle backOff event &TaskEx
it{ContainerID:82ab407df047fefdc54d0b4ef33558a456e67d66d99c2c727c96de84feee58ce,
ID:82ab407df047fefdc54d0b4ef33558a456e67d66d99c2c727c96de84feee58ce,Pid:8443,Exi
tStatus:143,ExitedAt:2020-06-01 08:37:03.665014952 +0000 UTC,XXX_unrecognized:[]
,} for 82ab407df047fefdc54d0b4ef33558a456e67d66d99c2c727c96de84feee58ce  error="
failed to handle container TaskExit event: failed to stop container: unknown err
or after kill: /home/suda/gopath/src/github.com/AkihiroSuda/critest-rootless-cgr
oup2/bin/runc did not terminate successfully: lstat /sys/fs/cgroup/user.slice/us
er-1001.slice/[email protected]/user.slice/cri-containerd-82ab407df047fefdc54d0b
4ef33558a456e67d66d99c2c727c96de84feee58ce.scope: no such file or directory\n: u
nknown"

repro

https://github.com/AkihiroSuda/critest-rootless-cgroup2/tree/v0.0.2

$ git clone https://github.com/AkihiroSuda/critest-rootless-cgroup2.git
$ cd critest-rootless-cgroup2
$ git checkout v0.0.2
$ ./bin-download.sh
$ cp $GOPATH/src/github.com/opencontainers/runc ./bin/runc
$ ./containerd.sh

Test with crun (succeeds):

$ ./run.sh crun
+ RUNTIME=crun
++ ./crictl.sh pods -q
+ [[ -n '' ]]
++ ./crictl.sh run --runtime=crun container-config.json pod-config.json
+ id=24c63a15fa18152acc378511c08680cc1383cfa1262e4fbb7688b858ce711e81
+ ./crictl.sh exec -it 24c63a15fa18152acc378511c08680cc1383cfa1262e4fbb7688b858ce711e81 sh
/ # exit
$ ./crictl.sh ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
24c63a15fa181       alpine              23 seconds ago      Running             my-container        0                   d466d29f9a286
$ ./crictl.sh stop 24c63a15fa181
24c63a15fa181

Test with runc (fails):

$ ./run.sh runc
+ RUNTIME=runc
++ ./crictl.sh pods -q
+ [[ -n '' ]]
++ ./crictl.sh run --runtime=runc container-config.json pod-config.json
+ id=82ab407df047fefdc54d0b4ef33558a456e67d66d99c2c727c96de84feee58ce
+ ./crictl.sh exec -it 82ab407df047fefdc54d0b4ef33558a456e67d66d99c2c727c96de84feee58ce sh
/ # exit
$ ./crictl.sh ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
82ab407df047f       alpine              6 seconds ago       Running             my-container        0                   cd300b9be02d8
$ ./crictl.sh stop 82ab407df047f 
(no output)

[cyphar]

Wouldn't hurt to do bpftrace -e 'kprobe:do_rmdir { printf("%s[%d] rmdir(%d, '%s')\n", comm, tid, arg0, str(arg1)); }' to find out what is removing the cgroup path.

[AkihiroSuda]

@cyphar
systemd seems deleting the dir when crictl stop was executed.

systemd[1091] rmdir(-100, /sys/fs/cgroup/user.slice/user-1001.slice/[email protected]/user.slice/cri-containerd-1ff0f143029e23b78b8a80d7df00ef90dcafd2c75758cc56d98580b4e2cc9298.scope)

[cyphar]

Yeah that was my hunch. It might be that systemd detects the cgroup is empty and auto-removes it (not sure if that's a thing) or we call StopUnit and systemd decides to remove the cgroup for us. Does this happen with the systemd cgroup driver or the cgroupfs one?

[AkihiroSuda]

systemd

[cyphar]

My guess is this is where the issue is coming from (from libcontainer/cgroups/systemd/v2.go in Destroy). Maybe we should just do os.Remove and ignore any errors.

    // XXX this is probably not needed, systemd should handle it
    err = os.Remove(m.path)
    if err != nil && !os.IsNotExist(err) {
        return err
    }

[AkihiroSuda]

The error seems happening during runc kill, not runc delete

panic: panic injected for debugging #2447: lstat /sys/fs/cgroup/user.slice/user-1001.slice/[email protected]/user.slice/cri-containerd-0435469eb20e01200333e2e13d94997b1b879a7755ec6c942e4dd86b800c0e10.scope: no such file or directory

goroutine 1 [running]:
github.com/opencontainers/runc/libcontainer/cgroups/systemd.(*unifiedManager).GetAllPids(0xc0002271a0, 0x55e400000003, 0xc0000df4d0, 0x1, 0x1, 0x0)
        /home/suda/gopath/src/github.com/opencontainers/runc/libcontainer/cgroups/systemd/v2.go:300 +0x105
github.com/opencontainers/runc/libcontainer.signalAllProcesses(0x55e44d0ac7e0, 0xc0002271a0, 0x55e44d0a3940, 0xc00022bdd8, 0xc0000df590, 0x55e44cad9a1d)
        /home/suda/gopath/src/github.com/opencontainers/runc/libcontainer/init_linux.go:479 +0xe4
github.com/opencontainers/runc/libcontainer.(*linuxContainer).Signal(0xc000246000, 0x55e44d0a3940, 0xc00022bdd8, 0x1, 0x0, 0x0)
        /home/suda/gopath/src/github.com/opencontainers/runc/libcontainer/container_linux.go:389 +0x299
main.glob..func7(0xc00020a160, 0x0, 0xc000098e70)
        /home/suda/gopath/src/github.com/opencontainers/runc/kill.go:54 +0x1aa
github.com/urfave/cli.HandleAction(0x55e44cff5240, 0x55e44d092780, 0xc00020a160, 0xc00020a160, 0x0)
        /home/suda/gopath/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:523 +0x11c
github.com/urfave/cli.Command.Run(0x55e44cef39c2, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55e44cf108b8, 0x52, 0x0, ...)
        /home/suda/gopath/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go:174 +0x570
github.com/urfave/cli.(*App).Run(0xc0000cc000, 0xc0000200c0, 0xc, 0xc, 0x0, 0x0)
        /home/suda/gopath/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:276 +0x7b0
main.main()
        /home/suda/gopath/src/github.com/opencontainers/runc/main.go:151 +0xb29

diff --git a/libcontainer/cgroups/systemd/v2.go b/libcontainer/cgroups/systemd/v2.go
index e78de667..5804a1c6 100644
--- a/libcontainer/cgroups/systemd/v2.go
+++ b/libcontainer/cgroups/systemd/v2.go
@@ -295,7 +295,11 @@ func (m *unifiedManager) GetAllPids() ([]int, error) {
        if err := m.initPath(); err != nil {
                return nil, err
        }
-       return cgroups.GetAllPids(m.path)
+       pids, err := cgroups.GetAllPids(m.path)
+       if err != nil {
+               panic(errors.Wrap(err, "panic injected for debugging #2447"))
+       }
+       return pids, err
 }

 func (m *unifiedManager) GetStats() (*cgroups.Stats, error) {

[lifubang]

maybe related to #2337

[AkihiroSuda]

Thanks @lifubang , yes, this is a dupe of #2337 and will be fixed by https://github.com/opencontainers/runc/pull/2338 .