K3s: Docker in docker setup constantly logs error

[ibuildthecloud]

You get the following error over and over again when running inside docker. The easiest way to see this is just run docker-compose up.

node_1    | E0223 01:01:20.444373       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f62bb7a8c33378725a935accb46039fd4d8dd3917f144f44a79aa052117f97c/kube-proxy": failed to get cgroup stats for "/docker/8f62bb7a8c33378725a935accb46039fd4d8dd3917f144f44a79aa052117f97c/kube-proxy": failed to get container info for "/docker/8f62bb7a8c33378725a935accb46039fd4d8dd3917f144f44a79aa052117f97c/kube-proxy": unknown container "/docker/8f62bb7a8c33378725a935accb46039fd4d8dd3917f144f44a79aa052117f97c/kube-proxy"

I believe this because of the custom changes we made to cadvisor. We need to revert those.

[ndemoor]

I've been trying to run k3s in https://labs.play-with-docker.com, which creates 'nodes' with d-in-d and can confirm the issue to loop on the agent nodes.

[everflux]

Looks to me like the kube proxy crashing immediately/ not running
There is not log message about the „startup cry“ so at least kube proxy does not finish startup imho.

[deniseschannon]

@tfiduccia can you test with v0.4.0 cause @ibuildthecloud says it might already be fixed

[everflux]

I can still reproduce the problem with 0.4.0

[tfiduccia]

Version - master 4/18
This is not fixed.
node_1 | E0418 19:44:42.171004 1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/7be6fa3ddd65c1059ee34a48321d63899aadfae3922ccd24b129d8972faa5655/kube-proxy": failed to get cgroup stats for "/docker/7be6fa3ddd65c1059ee34a48321d63899aadfae3922ccd24b129d8972faa5655/kube-proxy": failed to get container info for "/docker/7be6fa3ddd65c1059ee34a48321d63899aadfae3922ccd24b129d8972faa5655/kube-proxy": unknown container "/docker/7be6fa3ddd65c1059ee34a48321d63899aadfae3922ccd24b129d8972faa5655/kube-proxy"

[webwurst]

@ibuildthecloud Just in case you haven't seen this yet https://github.com/rancher/k3s/issues/87#issuecomment-486009796 Maybe a helpful finding to get this solved.

[nicosto]

Hi, thanks for this great job
this bug still present in v0.6.0-rc3

[datamgmt]

This issue has been around for a while now. Today I tried a clean install and used the latest versions

I used the latest file from:
https://github.com/rancher/k3s/blob/master/docker-compose.yml

I also upgraded the image to image: rancher/k3s:v0.6.1
I also tried removing --disable-agent adding the tmpfs and privileged as suggested elsewhere

I ended up with:

version: '3'
services:
  server:
    image: rancher/k3s:v0.6.1
    tmpfs:
    - /run
    - /var/run
    privileged: true
    command: server
    environment:
    - K3S_CLUSTER_SECRET=somethingtotallyrandom
    - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    - K3S_KUBECONFIG_MODE=666
    volumes:
    - k3s-server:/var/lib/rancher/k3s
    - .:/output
    ports:
    - 6443:6443

  node:
    image: rancher/k3s:v0.6.1
    tmpfs:
    - /run
    - /var/run
    privileged: true
    depends_on:
    - server
    environment:
    - K3S_URL=https://pi99:6443
    - K3S_CLUSTER_SECRET=somethingtotallyrandom

volumes:
  k3s-server: {}

In all cases I continue to get:

server_1  | E0621 11:28:49.435389       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/7e97344fc37ce943d4558c09426b022113cff8b1ae21f54900c31c6ab6ee6782": failed to get cgroup stats for "/docker/7e97344fc37ce943d4558c09426b022113cff8b1ae21f54900c31c6ab6ee6782": failed to get container info for "/docker/7e97344fc37ce943d4558c09426b022113cff8b1ae21f54900c31c6ab6ee6782": unknown container "/docker/7e97344fc37ce943d4558c09426b022113cff8b1ae21f54900c31c6ab6ee6782"
node_1    | E0621 11:28:50.675223       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/4ecd9d95b647f3023259911617eb9dd21d3279c5118950faaf8d0dd41bfd62c3": failed to get cgroup stats for "/docker/4ecd9d95b647f3023259911617eb9dd21d3279c5118950faaf8d0dd41bfd62c3": failed to get container info for "/docker/4ecd9d95b647f3023259911617eb9dd21d3279c5118950faaf8d0dd41bfd62c3": unknown container "/docker/4ecd9d95b647f3023259911617eb9dd21d3279c5118950faaf8d0dd41bfd62c3"

[datamgmt]

Some further research.

I found this blog post that had said it was working
https://www.mjpitz.com/blog/2019/04/10/k8s-k3s-rpi-oh-my/

• notable that it was an arm64/ubuntu and with v0.3.0 so I tried various combinations

The issue occurs with version 0.3.0 onward, and it occurs with armhf or arm64

It will run in docker if it is run serperately as follows

docker volume create k3s-server-vol

docker run -d \
  --name "k3s-server-ctr" \
  --network "host" \
  --restart "always" \
  -e "K3S_CLUSTER_SECRET=${SECRET}" \
  -v "k3s-server-vol:/var/lib/rancher/k3s" \
  -p "${PORT}:${PORT}" \
  rancher/k3s:${RELEASE} \
  server --disable-agent

docker run -d \
  --name "k3s-agent-ctr" \
  --network "host" \
  --restart "always" \
  -e "K3S_CLUSTER_SECRET=${SECRET}" \
  -e "K3S_URL=https://${MASTER}:${PORT}" \
  --privileged \
  --tmpfs /run \
  --tmpfs /var/run \
  rancher/k3s:${RELEASE}

In conclusion it appears to only be a problem if it is run from the docker-compose.yml but I can't find an obvious mistake in the docker-compose.yml

[outbackdingo]

same result for me, just tried master on debian under docker 18.09.7 continually scrolling message
node_3 | E0707 12:01:56.602576 1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/c88310cdb7eb512fe54f3dd1476e546eb10d1d024c604046eaa689ba4553071a/kube-proxy": failed to get cgroup stats for "/docker/c88310cdb7eb512fe54f3dd1476e546eb10d1d024c604046eaa689ba4553071a/kube-proxy": failed to get container info for "/docker/c88310cdb7eb512fe54f3dd1476e546eb10d1d024c604046eaa689ba4553071a/kube-proxy": unknown container "/docker/c88310cdb7eb512fe54f3dd1476e546eb10d1d024c604046eaa689ba4553071a/kube-proxy"
node_2 | E0707 12:01:56.643769 1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/25fc1eb7b7e541a81cbc52c27910acb3ef1129dcd9050b35960f298935a5547c/kube-proxy": failed to get cgroup stats for "/docker/25fc1eb7b7e541a81cbc52c27910acb3ef1129dcd9050b35960f298935a5547c/kube-proxy": failed to get container info for "/docker/25fc1eb7b7e541a81cbc52c27910acb3ef1129dcd9050b35960f298935a5547c/kube-proxy": unknown container "/docker/25fc1eb7b7e541a81cbc52c27910acb3ef1129dcd9050b35960f298935a5547c/kube-proxy"
node_1 | E0707 12:01:56.751434 1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/bdd26abd71c35127f48368a8556f3d5d1a461115bd4a8469ab2a877da73dc30d/kube-proxy": failed to get cgroup stats for "/docker/bdd26abd71c35127f48368a8556f3d5d1a461115bd4a8469ab2a877da73dc30d/kube-proxy": failed to get container info for "/docker/bdd26abd71c35127f48368a8556f3d5d1a461115bd4a8469ab2a877da73dc30d/kube-proxy": unknown container "/docker/bdd26abd71c35127f48368a8556f3d5d1a461115bd4a8469ab2a877da73dc30d/kube-proxy"

[pdxjohnny]

I'm also getting this on v0.7.0-rc6 (via k3d 1.3.0-dev.0)

E0723 01:54:57.888871       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/0cb3ec542966f91f0d4f6dd810062ba031dbb76bdc87265e0ba07fdc542be8f2/kube-proxy": failed to get cgroup stats for "/docker/0cb3ec542966f91f0d4f6dd810062ba031dbb76bdc87265e0ba07fdc542be8f2/kube-proxy": failed to get container info for "/docker/0cb3ec542966f91f0d4f6dd810062ba031dbb76bdc87265e0ba07fdc542be8f2/kube-proxy": unknown container "/docker/0cb3ec542966f91f0d4f6dd810062ba031dbb76bdc87265e0ba07fdc542be8f2/kube-proxy"

[huapox]

How's it going?

[erikwilson]

Fixed with #705