Microk8s: High CPU usage by gvfs-udisks2-vo which is caused by microk8s

[khteh]

Please run microk8s.inspect and attach the generated tarball to this issue.
inspection-report-20190610_165704.tar.gz

We appreciate your feedback. Thank you for using microk8s.
top shows:

 2293 khteh     20   0  270176  26568   7776 R  56.7   0.1 198:27.51 gvfs-udisks2-vo                                                                                                                               

udisksctl monitor shows it is mainly by microk8s. What happen?

[ktsakalozos]

Hi @khteh

I do not see this over here. Can you help me reproduce the issue you are seeing? Is it possible for you to share all the manifests you apply and the addons you enable?

I see that some hours ago you deployed a redis cluster. Could that be related with the spike in the disk utilization?

[khteh]

Hi @ktsakalozos these are the add-ons that I have enabled:

$ microk8s.status
microk8s is running
addons:
linkerd: enabled
jaeger: enabled
rbac: disabled
prometheus: enabled
dns: enabled
fluentd: enabled
storage: enabled
gpu: disabled
registry: disabled
dashboard: enabled
ingress: enabled
metrics-server: enabled
istio: enabled

I delete the statefulsets of redis cluster and now gvfs-udisk2-vo is at 10% CPU consumption. Is it the redis or something else?

[tdickman]

I am also seeing this occur in my cluster. Interestingly I have been using the same set of applications, so this must either be caused by a change in microk8s or a change in ubuntu.

I'll see if I can get a simplified set of applications - right now I'm running a number of private applications.

[graste]

Maybe this helps: disabling "istio" lead to significantly less cpu usage in my case for gvfs-udisks2….

[tellisnz-shift]

I have this as well and don't have a stateful set for redis, but do for things like kafka, zookeeper, elasticsearch.

udisksctl monitor has a lot of this.

10:06:29.082: /org/freedesktop/UDisks2/block_devices/loop2: org.freedesktop.UDisks2.Filesystem: Properties Changed
  MountPoints:          /run/containerd/runc/k8s.io/ed49d3cf89a55d4c6501e207ba8aee230e1c379c2c696a7a35db77021f8f30ec/runc.BjKdmt
                        /snap/microk8s/671
10:06:29.107: /org/freedesktop/UDisks2/block_devices/loop2: org.freedesktop.UDisks2.Filesystem: Properties Changed
  MountPoints:          /snap/microk8s/671
10:06:32.095: /org/freedesktop/UDisks2/block_devices/loop2: org.freedesktop.UDisks2.Filesystem: Properties Changed
  MountPoints:          /run/containerd/runc/k8s.io/f456b4f044b2c99883107eba795ba6c396a2134c0d9e8bded53db9dd2bc18823/runc.TJpcMy
                        /snap/microk8s/671
10:06:32.108: /org/freedesktop/UDisks2/block_devices/loop2: org.freedesktop.UDisks2.Filesystem: Properties Changed
  MountPoints:          /snap/microk8s/671
10:06:35.934: /org/freedesktop/UDisks2/block_devices/loop2: org.freedesktop.UDisks2.Filesystem: Properties Changed
  MountPoints:          /run/containerd/runc/k8s.io/ed49d3cf89a55d4c6501e207ba8aee230e1c379c2c696a7a35db77021f8f30ec/runc.kAhFaH
                        /snap/microk8s/671
10:06:35.956: /org/freedesktop/UDisks2/block_devices/loop2: org.freedesktop.UDisks2.Filesystem: Properties Changed
  MountPoints:          /snap/microk8s/671

The loop2 mount is the snap mount for microk8s.

/dev/loop2 /snap/microk8s/671 squashfs ro,nodev,relatime 0 0

Reading up about gvfs-udisks2-volume-monitor it seems there is the 'x-gvfs-hide' option you can provide when mounting but i don't know if that is like a filter for the events or just forces hiding in the gnome ui. Anyone got any ideas on how I can see by trying to add it to the snap mount?

Either way - losing half to one CPU on seemingly regular file ops containers are doing for a monitor that seems to not do much except (in my naive understanding of it) expose volumes/filesystem to GNOME is a heavy price to pay?

[ktsakalozos]

@tellisnz-shift just ask on the snapcraft forum [1] about the 'x-gvfs-hide' option. Perhaps the good people at snapcraft will be able to add this option when mounting MicroK8s. I see however that the same issue is reported for K3s [2] and the investigation over there pointed to outdated gnome packages. What distribution are you on? It would be great if we had a script that would reproduce this behavior.

[1] https://github.com/ubuntu/microk8s/issues/500#issuecomment-508609860
[2] https://github.com/rancher/k3s/issues/522

[tellisnz-shift]

Thanks for asking on there @ktsakalozos.

This is me currently:

cat /etc/os-release

NAME="Linux Mint"
VERSION="19 (Tara)"
ID=linuxmint
ID_LIKE=ubuntu
PRETTY_NAME="Linux Mint 19"
VERSION_ID="19"
...
VERSION_CODENAME=tara
UBUNTU_CODENAME=bionic

I'm fully up to date as far as I am aware. As far as scripting, I will try and find some time over the weekend.

[AdamIsrael]

I'm seeing the same behavior, on Ubuntu 19.04.

$ microk8s.status
microk8s is running
addons:
linkerd: disabled
jaeger: disabled
rbac: disabled
prometheus: enabled
dns: enabled
fluentd: disabled
storage: enabled
gpu: disabled
registry: disabled
knative: disabled
dashboard: enabled
ingress: disabled
metrics-server: disabled
istio: disabled

A few data points:

• I don't have redis running, but I do have kafka and zookeeper, among others, using persistent volumes. I removed each application, one by one, observing only marginal decreases in gvfs-udisks2-volume-monitor.

• It still had spikes of high CPU usage when I'd removed all applications. Disabling the storage plugin resolved it.

• I then re-enabled the storage plugin. There was a brief spike in CPU as the initial storage was created, and then it remained idle.

It makes sense that it's related to the storage plugin. The fact I saw high CPU usage after removing my services, and saw it resume to low CPU usage after enable/disabling the plugin, makes me think that there's something with the storage plugin that might be at fault.

[Grexeo]

I'm seeing the same, Ubuntu 18.04 LTS.

$ microk8s.status
microk8s is running
addons:
knative: disabled
jaeger: disabled
fluentd: disabled
gpu: disabled
cilium: disabled
storage: enabled
registry: disabled
rbac: disabled
ingress: enabled
dns: enabled
metrics-server: disabled
linkerd: disabled
prometheus: disabled
istio: disabled
dashboard: enabled

This issue locks up my high-spec development machine for a few minutes. I/O becomes saturated. This is with a very basic cluster deployment with no "traffic".

Interestingly, this only happens if I change network (i.e. connect to a wifi network) or resume from suspend. (Resuming from suspend may be triggering a "network change").

Disabling and re-enabling the storage plugin appears to fix it for a while. (As well as completely purging microK8s and reinstalling it)

[Multiply]

We're also seeing the same issue with Ubuntu.
The problem is less severe on computers with more cores, it seems.

[mariusstaicu]

on an older 4 core i7 cpu the freezes are almost unbearable

[tarun0]

I faced the similar issue after leaving the system idle. It froze almost every Monday. Switched to Minikube and it's better now.

[KeepMasterBranch]

Experience the same, and there is seem to be no way to disable it

[knkski]

I'm getting this issue when deploying Kubeflow, which has lots of pods that require storage. Since I'm not running the cluster full-time, I was able to work around this by running systemctl stop --user gvfs-udisks2-volume-monitor to disable the gvfs-udisks2-volume-monitor process.

Also, I tried adding the x-gvfs-hide option to all of the mounts, which didn't seem to help. udisksctl monitor showed very few events (unlike before adding the option), but gvfs-udisks2-volume-monitor still ate up CPU. It's possible that's due to me remounting with the flag instead of mounting with the flag. Here's what I ran to add the flag:

mount | grep /var/snap/microk8s | cut -d" " -f3 | xargs -I{} bash -c "echo {}; sudo mount -o remount,x-gvfs-hide {}"

[cameronbraid]

FYI I have this on a ubuntu 20.04 system with kubernetes installed using kubespray.

I did use microk8s at one point, but had already stopped it.

Interestingly if I run nautilus on this machine, it freezes. Same with all browser file save dialogs.

Running

systemctl stop --user gvfs-udisks2-volume-monitor

Fixes both the 100% CPU issue, and makes nautilus stop from freezing

[Richard87]

Same here to on Fedora 33... gvfs-udisks2-volume-monitor uses between 50 and 75% cpu constantly, and gsd-housekeeping is constantly between 25-50%

stopping monitor stopped it and its cpu usage, but I have no idea what side-effects that have?