Kops: Kubelet 'failed to get cgroup stats for "/system.slice/kubelet.service"' error messages

[jkemp101]

1. What kops version are you running? The command kops version, will display
   this information.
   Version 1.8.0 (git-5099bc5)

2. What Kubernetes version are you running? kubectl version will print the
   version if a cluster is running or provide the Kubernetes version specified as
   a kops flag.

Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.9-dirty", GitCommit:"7f63532e4ff4fbc7cacd96f6a95b50a49a2dc41b", GitTreeState:"dirty", BuildDate:"2017-10-26T22:33:15Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

1. What cloud provider are you using?
   AWS

2. What commands did you run? What is the simplest way to reproduce this issue?

Provisioned a 1.8.4 cluster by:

• Creating a basic cluster with kops
• Exporting that cluster to a manifest file
• Added my changes to manfiest
• Generate terraform configs

1. What happened after the commands executed?
   I get the following error message in the daemon logs unless I add the following to the manifest:

  kubelet:
    kubeletCgroups: "/systemd/system.slice"
    runtimeCgroups: "/systemd/system.slice"
  masterKubelet:
    kubeletCgroups: "/systemd/system.slice"
    runtimeCgroups: "/systemd/system.slice"

Dec 12 22:12:14 ip-172-20-64-61 kubelet[30742]: E1212 22:12:14.322073   30742 summary.go:92] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"

1. What did you expect to happen?
   No error messages

[chrislovecnm]

@blakebarnett ideas?

[blakebarnett]

Yes, dupe of https://github.com/kubernetes/kops/issues/1869

[jkemp101]

@blakebarnett This repeating error message about collecting stats is a duplicate of setting kubelet reserved resources?

[blakebarnett]

oh sorry, I misread the errors. I believe it's because the base image doesn't have proper cgroups configured though, which is ultimately the same problem as #1869

[jkemp101]

base image doesn't have proper cgroups configured

Well that doesn't sound good. I guess my first question is if the kubeletCgroups and runtimeCgroups settings I'm using is a safe workaround or is this a bigger issue? I got that fix from the stackoverflow thread mentioned in this issue https://github.com/kubernetes/kubernetes/issues/53960 but I didn't really research if this is a good fix or if I should be more concerned.

I should have mentioned the cluster was built with k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02

[chrislovecnm]

How about we close this and continue on the other issue?

[chrislovecnm]

Never mind the other issue is closed. See if I can find the one I am thinking of

[blakebarnett]

This one? https://github.com/kubernetes/kops/issues/3762

[chrislovecnm]

Closing as a duplicate of #3762

[jkemp101]

We can close this but I wouldn't call it a duplicate of #3762 unless you change the title. That issue is a recommendation to follow best practices, this is an error message. I'm trying to prepare new 1.8 clusters for critical production workloads. Can someone clarify if:

1. This error message is benign and should be ignored
2. The listed workaround using kubeletCgroups and runtimeCgroups should be used until #3762 is addressed.
3. We aren't sure what the impact of this error message is and therefore clusters experiencing this error message should probably not be used in a production environment.

[chrislovecnm]

/cc @itskingori @gambol99 @justinsb

Added a couple of gurus to see if they can help

[blakebarnett]

I think his solution of adding:

  kubelet:
    kubeletCgroups: "/systemd/system.slice"
    runtimeCgroups: "/systemd/system.slice"
  masterKubelet:
    kubeletCgroups: "/systemd/system.slice"
    runtimeCgroups: "/systemd/system.slice"

as a default seems sane, this slice does exist and may help provide some metrics (I'm not sure where they're exposed, or what we're missing without them yet)

[jkemp101]

Thanks for getting back to this quickly. Sounds like I should stick with my current workaround until other fixes are in place. This issue can be closed if there aren't any other suggestions.

[blakebarnett]

Let's keep it open until we either get the defaults / docs updated or come up with a better solution.

[mercantiandrea]

this workaround works also on AWS default image for kops (k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02 (ami-bd229ec4))

sudo vim /etc/sysconfig/kubelet

add at the end of DAEMON_ARGS string:

--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice

finally:

sudo systemctl restart kubelet

But I think that this problem should be fixed in the kops release

[jordanjennings]

Thanks for the workaround! One thing that I noticed it fixes is the reported docker version from kubectl get nodes -o wide

First node has the workaround, second doesn't:

NAME                             STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION   CONTAINER-RUNTIME
ip-10-244-115-162.ec2.internal   Ready     node      23m       v1.8.6    <none>        Debian GNU/Linux 8 (jessie)   4.4.102-k8s      docker://17.9.0
ip-10-244-117-51.ec2.internal    Ready     node      21h       v1.8.5    <none>        Debian GNU/Linux 8 (jessie)   4.4.102-k8s      docker://Unknown

[oliverseal]

I'm using terraform output to spin up several clusters on AWS and this has been a source of aggravation having to go to each instance and edit the /etc/sysconfig/kubelet file. It's made aws auto scaling groups almost useless.

[mercantiandrea]

@oliverseal You can modify the user-data bash script in the terraform template. It is encoded in base64 but you can easily decode, modify and finally encode again.

[oliverseal]

Added this to the userdata after "download-release":

sed -i 's@--network-plugin-dir=/opt/cni/bin/@--network-plugin-dir=/opt/cni/bin/ --runtime-cgroups=/systemd/system.s--kubelet-cgroups=/systemd/system.slice@' /etc/sysconfig/kubelet\"
systemctl restart kubelet

and it seems to be working fine.

[itskingori]

@mercantiandrea @oliverseal Seems you're doing manually what's done automatically for you, i.e. https://github.com/kubernetes/kops/issues/4049#issuecomment-352152838. You only need to add these when you edit your cluster:

spec:
  kubelet:
    kubeletCgroups: "/systemd/system.slice"
    runtimeCgroups: "/systemd/system.slice"
  masterKubelet:
    kubeletCgroups: "/systemd/system.slice"
    runtimeCgroups: "/systemd/system.slice"

[igoratencompass]

@itskingori the point is this should be the default settings instead of us going in the config and making changes since we know default values are wrong. One by one things like this add up and you end up with a long list of custom settings you need to apply every time you create a cluster.

[SpComb]

See https://github.com/kontena/pharos-cluster/issues/440#issuecomment-399014418 for why the --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice workaround is a bad idea on CentOS: the extra /systemd prefix causes the kubelet and dockerd processes to escape from their correct systemd cgroups into a new /systemd/system.slice cgroup created next to the real /system.slice cgroup.

I'm not entirely sure how much the systemd cgroup names differ across OSes, but I assume that the workaround was actually meant to be --runtime-cgroups=/system.slice --kubelet-cgroups=/system.slice... that's only slightly better https://github.com/kontena/pharos-cluster/issues/440#issuecomment-399017863: the processes still escape from the systemd kubelet.service / docker.service cgropus, and the kubelet /stats/summary API still reports the wrong numbers.

The correct fix is to enable systemd CPUAccounting and MemoryAccounting for the kubelet.service... this causes systemd to create missing the /system.slice/*.service cgroups for all services, and matches what happens by default on e.g. Ubuntu xenial. This allows the kubelet /stats/summary API to report the correct systemContainer metrics for the runtime and kubelet: https://github.com/kontena/pharos-cluster/issues/440#issuecomment-399022473

I think these systemd settings should be shipped as part of the upstream kubelet package's kubelet.service, if the kubelet assumes that systemd creates those cgroups?

/etc/systemd/system/kubelet.service.d/11-cgroups.conf

[Service]
CPUAccounting=true
MemoryAccounting=true

[justinsb]

We should fix this in 1.11

[Guezennec]

hi all
not fixed in 1.11 :-( , just setup release 1.11 and still have the bug , the config file has changed it's now a yaml file , where we can add the workaround thanks

[tsuna]

After applying the workaround I now see this seemingly useless log message every 5min:

container_manager_linux.go:427] [ContainerManager]: Discovered runtime cgroups name: /systemd/system.slice

[rekcah78]

@Guezennec maybe @justinsb says kops 1.11 not k8s 1.11, and kops 1.11 is not yet released.

[mmerrill3]

I added this to my PR since I was in the same area: https://github.com/kubernetes/kops/pull/5939

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[tsuna]

Not sure if this expected but I still see this every 5min, although the workaround is not needed anymore (am on k8s 1.11.7 now with kops 1.12-alpha):

I0212 22:31:26.146763    3334 container_manager_linux.go:428] [ContainerManager]: Discovered runtime cgroups name: /systemd/system.slice

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.