Version: k3s version v0.10.2 (8833bfd9)
Describe the bug
After running for +/- 3 days the host becomes unavailable. To many open files.
Now running for 1 day I have 1250597 open inodes.
$ lsof
...
container 5747 25760 container root *040u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *041u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *042u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *043u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *044u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *045u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *046u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *047u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *048u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *049u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *050u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *051u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *052u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *053u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *054u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *055u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *056u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *057u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *058u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *059u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *060u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *061u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *062u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *063u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *064u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *065u a_inode 0,13 0 9259 [eventfd]
container 5747 25760 container root *066u a_inode 0,13 0 9259 [eventfd]
...
To Reproduce
Run pods not sure how to reproduce. It started to appear from version upgrade 0.9 to 0.10
Expected behavior
Actual behavior
Host runs out nodes.
kinupstream-issue
johansmitsnl
All 24 comments
Now 24 hour later the count is at: 2130156
johansmitsnl
on 12 Nov 2019
Could you give us some more info about your cluster?
Also, how many nodes? How many pods running?
Edit: Also, output of df -i might help.
davidnuzik
on 12 Nov 2019
@davidnuzik sure:
The custer is part of 2 nodes with ipsec as the backend. Perhaps unrelated but I have also this issue: #951
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
s01-01 Ready <none> 10d v1.16.2-k3s.1
s00 Ready master 179d v1.16.2-k3s.1
Running nodes on the s01-01:
$ kubectl get pods -o wide --sort-by="{.spec.nodeName}" --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
longhorn-system engine-image-ei-3827e67c-dkd5r 1/1 Running 2 10d 10.42.1.98 s01-01 <none> <none>
longhorn-system longhorn-driver-deployer-ddfcc8b5b-pmrjd 1/1 Running 1 10d 10.42.1.107 s01-01 <none> <none>
longhorn-system instance-manager-e-2f2e5328 1/1 Running 0 2d23h 10.42.1.118 s01-01 <none> <none>
longhorn-system instance-manager-r-d40ff6bd 1/1 Running 0 2d23h 10.42.1.119 s01-01 <none> <none>
infrastructure gitlab-runner-84c697d85f-8w5ws 1/1 Running 1 7d12h 10.42.1.112 s01-01 <none> <none>
longhorn-system longhorn-manager-wwdrk 1/1 Running 1 10d 10.42.1.105 s01-01 <none> <none>
cameras s01-c05-86dd7c7c45-rjws6 1/1 Running 1 10d 10.42.1.101 s01-01 <none> <none>
cameras s01-rec-7dc9f4759c-hwc58 1/1 Running 1 10d 10.42.1.117 s01-01 <none> <none>
longhorn-system csi-provisioner-55468c9577-zh4qx 1/1 Running 1 10d 10.42.1.106 s01-01 <none> <none>
cameras s01-c04-655c4d98f9-6vpt4 1/1 Running 1 10d 10.42.1.116 s01-01 <none> <none>
infrastructure ldapadmin-6479f75dcd-cvccl 1/1 Running 1 10d 10.42.1.100 s01-01 <none> <none>
home svclb-s01-plex-media-server-external-hk9mz 1/1 Running 1 3d14h 10.42.1.114 s01-01 <none> <none>
home svclb-s01-assistant-homekit-q8kmh 1/1 Running 1 7d10h 10.42.1.113 s01-01 <none> <none>
cameras s01-c01-5847579465-v7tg9 1/1 Running 1 10d 10.42.1.115 s01-01 <none> <none>
longhorn-system csi-attacher-748fdbdc64-ldfhd 1/1 Running 10 10d 10.42.1.99 s01-01 <none> <none>
kube-system svclb-traefik-9nzgn 3/3 Running 3 10d 10.42.1.108 s01-01 <none> <none>
longhorn-system csi-attacher-748fdbdc64-hf86t 1/1 Running 10 10d 10.42.1.111 s01-01 <none> <none>
longhorn-system csi-provisioner-55468c9577-2fjn9 1/1 Running 1 10d 10.42.1.104 s01-01 <none> <none>
cameras s01-c03-59d95968b9-km7g2 1/1 Running 1 10d 10.42.1.103 s01-01 <none> <none>
longhorn-system longhorn-ui-659687f745-dvxfn 1/1 Running 14 10d 10.42.1.109 s01-01 <none> <none>
longhorn-system longhorn-csi-plugin-dtfg6 2/2 Running 25 10d 10.42.1.110 s01-01 <none> <none>
infrastructure gitlab-5466568b88-4k6zg 1/1 Running 0 2d22h 10.42.1.120 s01-01 <none> <none>
home s01-assistant-66cd56b8f8-zpm2z 2/2 Running 0 2d22h 10.42.1.121 s01-01 <none> <none>
home s01-plex-media-server-8588bd4654-plrnl 5/5 Running 0 2d22h 10.42.1.122 s01-01 <none> <none>
cameras s01-c02-584fcbbc95-8zjl2 1/1 Running 1 10d 10.42.1.102 s01-01 <none> <none>
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 490444 440 490004 1% /dev
tmpfs 494156 1543 492613 1% /run
/dev/sda2 1667904 584085 1083819 36% /
tmpfs 494156 1 494155 1% /dev/shm
tmpfs 494156 4 494152 1% /run/lock
tmpfs 494156 17 494139 1% /sys/fs/cgroup
/dev/sda1 0 0 0 - /boot/efi
/dev/sdb1 488363520 78 488363442 1% /srv/bigdisk1
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/6354ea85-7c46-4a4c-bb87-a7fc8fcc10fc/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/75dccbd1-0a39-42a5-beea-0773e1aac39b/volumes/kubernetes.io~secret/default-token-nz7vt
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/19cdaa1b-f834-4b9a-8189-03f281e79f1c/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/63ebe2e8-46e8-4e26-8d6d-19371c255a84/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/e1b0e8d9-e912-4a24-b5f7-8294c151a919/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/9037206a-76e2-4440-b90f-d0efa00db2a8/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/64d92205-6d01-420f-bae4-6522bef1fcbb/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/74bc13bf-7e1a-4b7d-9c96-513c8804684d/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/39ad503f-ae54-4541-a380-46e08932f8ef/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/d9a086f2-dda9-4d2f-8a49-d5768a9b6328/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/79639802-a528-4b7a-a0a8-fed2cb29b4a8/volumes/kubernetes.io~secret/default-token-dcqr9
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/95400d02-c111-4556-bafb-d343c578aa00/volumes/kubernetes.io~secret/default-token-dcqr9
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/7bc4201151c2af51986b85c603ff7acaf7f4f9c508e812143c6eced0f6fb664a/shm
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/b006272a-bfe2-4105-99bd-9a38bdad60ea/volumes/kubernetes.io~secret/default-token-jgf6h
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/1d9192b0-c17b-4679-b99b-8ebefe8879d0/volumes/kubernetes.io~secret/default-token-jgf6h
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/32209efd-d4e2-4d89-90ed-0a57a47ca9df/volumes/kubernetes.io~secret/default-token-jgf6h
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/629bfa21cf536336df47019b7ff99862aab4d8624dcc178b8c5ecab6d435f2a9/shm
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/3aa0ac2839787c4d07bdd3ac46df1200df4d57f2c881d64dcb5d5c4cbbea6d95/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7bc4201151c2af51986b85c603ff7acaf7f4f9c508e812143c6eced0f6fb664a/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/629bfa21cf536336df47019b7ff99862aab4d8624dcc178b8c5ecab6d435f2a9/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/3aa0ac2839787c4d07bdd3ac46df1200df4d57f2c881d64dcb5d5c4cbbea6d95/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/42bbb1b3b0985e17b2f81abb6c38be66bb7271583fb0fdc191cc147a33b5bac0/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/42bbb1b3b0985e17b2f81abb6c38be66bb7271583fb0fdc191cc147a33b5bac0/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/9bab7eb7d43c8d01fc2d606c07a6cf69b54776ab9d9d3911ee1aa81935da05c7/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/9bab7eb7d43c8d01fc2d606c07a6cf69b54776ab9d9d3911ee1aa81935da05c7/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/7df7c5365e98aee214f3a61bc91c286581c487395cb70bb3ce41554b29719bf3/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7df7c5365e98aee214f3a61bc91c286581c487395cb70bb3ce41554b29719bf3/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/c8902c084d05c64dd33036b7de56ca056bdf47701c36f3f959b6bfbaa2ad4d44/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c8902c084d05c64dd33036b7de56ca056bdf47701c36f3f959b6bfbaa2ad4d44/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/4c8ec7a5046761be4839fcf267033aa06bacdfc8df31376f9c6a6373e72ba37f/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/4c8ec7a5046761be4839fcf267033aa06bacdfc8df31376f9c6a6373e72ba37f/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/829377a31d1679ce636484d411ef52d0d3cb5f335ede2fab9cfdd4da0dd64791/shm
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/a25c416170e79cb97f614fcfa166394dc0d5c773ab20d6971f1c01866bba0ee2/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/829377a31d1679ce636484d411ef52d0d3cb5f335ede2fab9cfdd4da0dd64791/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a25c416170e79cb97f614fcfa166394dc0d5c773ab20d6971f1c01866bba0ee2/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/a4c311c52286d16a6979fc4d1fb8a14b8808f7809af2ebcc899472483b91ad9b/shm
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/13d16b8872bec7b46c685f9537ff71d5a6e7a4a893c400be3375845f781cc8b6/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a4c311c52286d16a6979fc4d1fb8a14b8808f7809af2ebcc899472483b91ad9b/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/13d16b8872bec7b46c685f9537ff71d5a6e7a4a893c400be3375845f781cc8b6/rootfs
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/3cd91b4c-da0a-41cf-b99e-b0a3759d16de/volumes/kubernetes.io~secret/default-token-8thgc
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/6cdd3bba-f9ee-4bb8-8f64-0bbaf0da48af/volumes/kubernetes.io~secret/default-token-8thgc
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/8636f0967829d1a262562b80ca6029aca34924ad05e008cc425d775ce864859e/shm
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/a3f9e1ff85077f4d1c86444c4167841aa1a47cc427a8d146ed3e00deb635a669/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8636f0967829d1a262562b80ca6029aca34924ad05e008cc425d775ce864859e/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/d8cc26c47769504d85166f05dd1e62ded50019465162d7437b2a54685a956e06/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a3f9e1ff85077f4d1c86444c4167841aa1a47cc427a8d146ed3e00deb635a669/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d8cc26c47769504d85166f05dd1e62ded50019465162d7437b2a54685a956e06/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/23c45a4a7e55e97b5065f54c77b55b61e6889c67257ee7817fbcb7748828ab52/rootfs
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/bf6dd52a-5c6a-4afe-a67a-2b27b460fbce/volumes/kubernetes.io~secret/default-token-jgf6h
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/aac80d76-7c56-4179-91f3-eacfc626e752/volumes/kubernetes.io~secret/default-token-jgf6h
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/e69fd271-720b-4145-9bcc-083df6d10b5c/volumes/kubernetes.io~secret/default-token-jgf6h
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/6d753898950c88bc78fb1ed18559c3138ddc7f4825ead53b64509481240e5e20/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/6d753898950c88bc78fb1ed18559c3138ddc7f4825ead53b64509481240e5e20/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c484a4b35764206ec39cf64ccdb8314939e98c78fc8030894b7fbe9e32f73dac/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/072de44c6c940a4c610f1cc85a1b575864651a7feefe43543bd66d4589bc6c73/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/072de44c6c940a4c610f1cc85a1b575864651a7feefe43543bd66d4589bc6c73/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/9a35345137001d48b42077e458890d471a0dacdf43eed06b5b0a64a3cd4941dc/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/cdee4c2b932a7bd8db0bc3434c2ee1900b6dc746c4af690df75028f7a55dee9e/shm
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/1b76cbced455e650cd542e3fb278d69035e6d756518a52fee0579947e263df0d/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/cdee4c2b932a7bd8db0bc3434c2ee1900b6dc746c4af690df75028f7a55dee9e/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/77bced165842e53aa4b9739ff466d2e14c330b689b6f1f78818b661f9a94e61d/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/1b76cbced455e650cd542e3fb278d69035e6d756518a52fee0579947e263df0d/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/77bced165842e53aa4b9739ff466d2e14c330b689b6f1f78818b661f9a94e61d/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/dbbd21fe11ecd1644bae88f713e82b8be0af4d25b6ec882a477c85fc6782f682/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/ae236fd543b66fe6f2f4c4c5b1cd077363c52500bb7dd59ada7a67cfeacfd1d9/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d9783d687797d8b7f268685acc604a1dfd108c9991151cbbf6ee4d563394efbd/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a5130edfacf4105891cea50440f492f3ba36e4e463576f00ce913815ed152bfb/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a37524fabf3677ee1091ba910abfaff75a2952b6cb97c1a8ba2eac145adcd453/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/6c6f9232d6196ec087298ca7214ee998cd1cb2c94cc4b21ff2a835af411d7939/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7c95605e2971442165b968899b767c73b4a28ce4c37395212d48b4fdb5453230/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8316503cd342a19e8e916b6a861cc1efedce2432e357f4c4241db27c16c5cffc/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/257c990940df1fdff2accea03f5c5409111b6442cba3daa3bc1901c63cad4482/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e2120f58ea304621d3f376b85d0c011881c56ec66113e2d063dd12207df12264/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d945681df082d185b80b97b5173b8db9fe6cc141fc79d3c58288d57a7f48e47e/rootfs
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/7bd67ab2-8a0c-41c4-b503-47401ff201da/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/ef7a87d4b993d63c892f90d657f861c39c044c57f4cb722b6b00dd7e19181cdb/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/4b9ee8a4a12c85437a286ec6aa227bd701a1c62b8e4dc56b00bcee872cec7b6a/rootfs
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/faad50f2ae109a43f1985e718c5a0529a666768e7873ccb7d1caa887ef78ec03/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/faad50f2ae109a43f1985e718c5a0529a666768e7873ccb7d1caa887ef78ec03/rootfs
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/8f9a7664-0957-4531-945f-17a92c9b118d/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
shm 494156 1 494155 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/4089c8564ec555a2c80b985712582952527765caea5cf1b3dcde1ae3a6cd35a5/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/4089c8564ec555a2c80b985712582952527765caea5cf1b3dcde1ae3a6cd35a5/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/14ba389acba26efa0d64dfdc358b563411e53718844c44fe1269d20afee32d4b/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/087fa46b30ee0c590f85d8cb0119608b36e56e0380571b37d67112d31fdc98c0/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d0af0fbcfa82766eae3a70df089ec9880016eac84f39e223e3f28e45ae44c862/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d5de87a8cd4302f0abd8373a5a706662eae7ae10283edec865b44399d9874c52/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/cb41b7f22ecfececd5e965f7abf3474d17aacefe71b3d837a9028a8f83840247/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d3e394544349a86dceaa96c717487a48a19d42416cd4b2496aab226a63cb8b61/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/0855b7b4660fda38fd44cfdf3cd102712dc40dd026e45d5a0c3d3d599ab3a393/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/706c8a2266735ffa43ad3e452e2aa82e26257b377d8c972c4913d0960c2176b3/rootfs
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/f9198705-cb41-48fe-a8a6-bc5b8377d704/volumes/kubernetes.io~secret/default-token-8thgc
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/f8990ed2-a75e-45d6-9093-d13ae79729dc/volumes/kubernetes.io~secret/certificates
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/f8990ed2-a75e-45d6-9093-d13ae79729dc/volumes/kubernetes.io~secret/default-token-dcqr9
tmpfs 494156 9 494147 1% /var/lib/kubelet/pods/761179a6-6f11-4611-816b-2c1da9814b54/volumes/kubernetes.io~secret/default-token-8thgc
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/3d77d740d56a396211a9818630ec91e7e766327d122458a51a07c77525cb0809/rootfs
/dev/longhorn/pvc-73b89aea-3338-4155-83af-e2626fb650ec 655360 48977 606383 8% /var/lib/kubelet/pods/f8990ed2-a75e-45d6-9093-d13ae79729dc/volumes/kubernetes.io~csi/pvc-73b89aea-3338-4155-83af-e2626fb650ec/mount
shm 494156 5 494151 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/158bae21bc1d83fe8c83316fc3edae0e81cd35fb295dcd06d4938c9957d947e8/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/158bae21bc1d83fe8c83316fc3edae0e81cd35fb295dcd06d4938c9957d947e8/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/b091e6e335a19a151df064c6bffe4a3a8c3b668c409dcb2703bd264371f5886d/rootfs
/dev/longhorn/pvc-77756bb4-21e4-4c05-a69c-c614e0ea1569 327680 10530 317150 4% /var/lib/kubelet/pods/761179a6-6f11-4611-816b-2c1da9814b54/volumes/kubernetes.io~csi/pvc-77756bb4-21e4-4c05-a69c-c614e0ea1569/mount
shm 494156 2 494154 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/623eeecd47a036b6e06d9170a6a9c386d0afe4c00c353cee85ea0f1d55912880/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/623eeecd47a036b6e06d9170a6a9c386d0afe4c00c353cee85ea0f1d55912880/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/2e44472347fcf458cf71eddb6b52983dcf483bcd4748ebceb9c789d3a614a3e7/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/f2d4be789af94b35f67c2e1fbf6d5d3d082e6ee8915a11eff6ba4fbda5d5873c/rootfs
/dev/longhorn/pvc-cedad9ef-aca3-4e6e-8df3-e3f6559952ab 6553600 5003 6548597 1% /var/lib/kubelet/pods/f9198705-cb41-48fe-a8a6-bc5b8377d704/volumes/kubernetes.io~csi/pvc-cedad9ef-aca3-4e6e-8df3-e3f6559952ab/mount
shm 494156 4 494152 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/e1ed31e97f4c7358dbf4352437a5aa14230bff0b40fff95b08812319b88738cb/shm
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e1ed31e97f4c7358dbf4352437a5aa14230bff0b40fff95b08812319b88738cb/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/87f869d31a718fe191476d540b37f67b6575115d43421eb7a3e62bfe6b4a90d8/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8d4dce4511018d9e1cb3f1cd0d9049de402d25d0342e07353d46f313f7309a30/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/94b4fd3fda664f68819d1fce9e8da0bacf4d8a31949402cb64575e138222e38e/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7074fedc3ebfad9ecd242fd5232b0edcab51154baec7cd3943ba64e3b8411e72/rootfs
overlay 1667904 584085 1083819 36% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/5cbca1ee8858615cf47b81c002d3ea758ce2c4616699d23609ea7b52e9cbb34e/rootfs
tmpfs 494156 4 494152 1% /run/user/0
Now the count is at: 2618743
johansmitsnl
on 13 Nov 2019
Hey @erikwilson can you take a look at this
cjellick
on 13 Nov 2019
Now the output is like:
$ df -i
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 387M 42M 345M 11% /run
/dev/sda2 25G 17G 6.9G 72% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 511M 10M 502M 2% /boot/efi
/dev/sdb1 932G 63G 869G 7% /srv/bigdisk1
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/6354ea85-7c46-4a4c-bb87-a7fc8fcc10fc/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/75dccbd1-0a39-42a5-beea-0773e1aac39b/volumes/kubernetes.io~secret/default-token-nz7vt
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/19cdaa1b-f834-4b9a-8189-03f281e79f1c/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/63ebe2e8-46e8-4e26-8d6d-19371c255a84/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/e1b0e8d9-e912-4a24-b5f7-8294c151a919/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/9037206a-76e2-4440-b90f-d0efa00db2a8/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/64d92205-6d01-420f-bae4-6522bef1fcbb/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/74bc13bf-7e1a-4b7d-9c96-513c8804684d/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/39ad503f-ae54-4541-a380-46e08932f8ef/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/d9a086f2-dda9-4d2f-8a49-d5768a9b6328/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/79639802-a528-4b7a-a0a8-fed2cb29b4a8/volumes/kubernetes.io~secret/default-token-dcqr9
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/95400d02-c111-4556-bafb-d343c578aa00/volumes/kubernetes.io~secret/default-token-dcqr9
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/7bc4201151c2af51986b85c603ff7acaf7f4f9c508e812143c6eced0f6fb664a/shm
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/b006272a-bfe2-4105-99bd-9a38bdad60ea/volumes/kubernetes.io~secret/default-token-jgf6h
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/1d9192b0-c17b-4679-b99b-8ebefe8879d0/volumes/kubernetes.io~secret/default-token-jgf6h
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/32209efd-d4e2-4d89-90ed-0a57a47ca9df/volumes/kubernetes.io~secret/default-token-jgf6h
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/629bfa21cf536336df47019b7ff99862aab4d8624dcc178b8c5ecab6d435f2a9/shm
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/3aa0ac2839787c4d07bdd3ac46df1200df4d57f2c881d64dcb5d5c4cbbea6d95/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7bc4201151c2af51986b85c603ff7acaf7f4f9c508e812143c6eced0f6fb664a/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/629bfa21cf536336df47019b7ff99862aab4d8624dcc178b8c5ecab6d435f2a9/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/3aa0ac2839787c4d07bdd3ac46df1200df4d57f2c881d64dcb5d5c4cbbea6d95/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/42bbb1b3b0985e17b2f81abb6c38be66bb7271583fb0fdc191cc147a33b5bac0/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/42bbb1b3b0985e17b2f81abb6c38be66bb7271583fb0fdc191cc147a33b5bac0/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/9bab7eb7d43c8d01fc2d606c07a6cf69b54776ab9d9d3911ee1aa81935da05c7/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/9bab7eb7d43c8d01fc2d606c07a6cf69b54776ab9d9d3911ee1aa81935da05c7/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/7df7c5365e98aee214f3a61bc91c286581c487395cb70bb3ce41554b29719bf3/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7df7c5365e98aee214f3a61bc91c286581c487395cb70bb3ce41554b29719bf3/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/c8902c084d05c64dd33036b7de56ca056bdf47701c36f3f959b6bfbaa2ad4d44/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c8902c084d05c64dd33036b7de56ca056bdf47701c36f3f959b6bfbaa2ad4d44/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/4c8ec7a5046761be4839fcf267033aa06bacdfc8df31376f9c6a6373e72ba37f/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/4c8ec7a5046761be4839fcf267033aa06bacdfc8df31376f9c6a6373e72ba37f/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/829377a31d1679ce636484d411ef52d0d3cb5f335ede2fab9cfdd4da0dd64791/shm
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/a25c416170e79cb97f614fcfa166394dc0d5c773ab20d6971f1c01866bba0ee2/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/829377a31d1679ce636484d411ef52d0d3cb5f335ede2fab9cfdd4da0dd64791/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a25c416170e79cb97f614fcfa166394dc0d5c773ab20d6971f1c01866bba0ee2/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/a4c311c52286d16a6979fc4d1fb8a14b8808f7809af2ebcc899472483b91ad9b/shm
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/13d16b8872bec7b46c685f9537ff71d5a6e7a4a893c400be3375845f781cc8b6/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a4c311c52286d16a6979fc4d1fb8a14b8808f7809af2ebcc899472483b91ad9b/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/13d16b8872bec7b46c685f9537ff71d5a6e7a4a893c400be3375845f781cc8b6/rootfs
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/3cd91b4c-da0a-41cf-b99e-b0a3759d16de/volumes/kubernetes.io~secret/default-token-8thgc
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/6cdd3bba-f9ee-4bb8-8f64-0bbaf0da48af/volumes/kubernetes.io~secret/default-token-8thgc
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/8636f0967829d1a262562b80ca6029aca34924ad05e008cc425d775ce864859e/shm
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/a3f9e1ff85077f4d1c86444c4167841aa1a47cc427a8d146ed3e00deb635a669/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8636f0967829d1a262562b80ca6029aca34924ad05e008cc425d775ce864859e/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/d8cc26c47769504d85166f05dd1e62ded50019465162d7437b2a54685a956e06/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a3f9e1ff85077f4d1c86444c4167841aa1a47cc427a8d146ed3e00deb635a669/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d8cc26c47769504d85166f05dd1e62ded50019465162d7437b2a54685a956e06/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/23c45a4a7e55e97b5065f54c77b55b61e6889c67257ee7817fbcb7748828ab52/rootfs
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/bf6dd52a-5c6a-4afe-a67a-2b27b460fbce/volumes/kubernetes.io~secret/default-token-jgf6h
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/aac80d76-7c56-4179-91f3-eacfc626e752/volumes/kubernetes.io~secret/default-token-jgf6h
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/e69fd271-720b-4145-9bcc-083df6d10b5c/volumes/kubernetes.io~secret/default-token-jgf6h
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/6d753898950c88bc78fb1ed18559c3138ddc7f4825ead53b64509481240e5e20/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/6d753898950c88bc78fb1ed18559c3138ddc7f4825ead53b64509481240e5e20/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c484a4b35764206ec39cf64ccdb8314939e98c78fc8030894b7fbe9e32f73dac/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/072de44c6c940a4c610f1cc85a1b575864651a7feefe43543bd66d4589bc6c73/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/072de44c6c940a4c610f1cc85a1b575864651a7feefe43543bd66d4589bc6c73/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/9a35345137001d48b42077e458890d471a0dacdf43eed06b5b0a64a3cd4941dc/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/cdee4c2b932a7bd8db0bc3434c2ee1900b6dc746c4af690df75028f7a55dee9e/shm
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/1b76cbced455e650cd542e3fb278d69035e6d756518a52fee0579947e263df0d/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/cdee4c2b932a7bd8db0bc3434c2ee1900b6dc746c4af690df75028f7a55dee9e/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/77bced165842e53aa4b9739ff466d2e14c330b689b6f1f78818b661f9a94e61d/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/1b76cbced455e650cd542e3fb278d69035e6d756518a52fee0579947e263df0d/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/77bced165842e53aa4b9739ff466d2e14c330b689b6f1f78818b661f9a94e61d/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/dbbd21fe11ecd1644bae88f713e82b8be0af4d25b6ec882a477c85fc6782f682/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/ae236fd543b66fe6f2f4c4c5b1cd077363c52500bb7dd59ada7a67cfeacfd1d9/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d9783d687797d8b7f268685acc604a1dfd108c9991151cbbf6ee4d563394efbd/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a5130edfacf4105891cea50440f492f3ba36e4e463576f00ce913815ed152bfb/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/a37524fabf3677ee1091ba910abfaff75a2952b6cb97c1a8ba2eac145adcd453/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/6c6f9232d6196ec087298ca7214ee998cd1cb2c94cc4b21ff2a835af411d7939/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7c95605e2971442165b968899b767c73b4a28ce4c37395212d48b4fdb5453230/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8316503cd342a19e8e916b6a861cc1efedce2432e357f4c4241db27c16c5cffc/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/257c990940df1fdff2accea03f5c5409111b6442cba3daa3bc1901c63cad4482/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e2120f58ea304621d3f376b85d0c011881c56ec66113e2d063dd12207df12264/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d945681df082d185b80b97b5173b8db9fe6cc141fc79d3c58288d57a7f48e47e/rootfs
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/7bd67ab2-8a0c-41c4-b503-47401ff201da/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/ef7a87d4b993d63c892f90d657f861c39c044c57f4cb722b6b00dd7e19181cdb/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/4b9ee8a4a12c85437a286ec6aa227bd701a1c62b8e4dc56b00bcee872cec7b6a/rootfs
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/faad50f2ae109a43f1985e718c5a0529a666768e7873ccb7d1caa887ef78ec03/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/faad50f2ae109a43f1985e718c5a0529a666768e7873ccb7d1caa887ef78ec03/rootfs
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/8f9a7664-0957-4531-945f-17a92c9b118d/volumes/kubernetes.io~secret/longhorn-service-account-token-76c22
shm 64M 0 64M 0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/4089c8564ec555a2c80b985712582952527765caea5cf1b3dcde1ae3a6cd35a5/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/4089c8564ec555a2c80b985712582952527765caea5cf1b3dcde1ae3a6cd35a5/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/14ba389acba26efa0d64dfdc358b563411e53718844c44fe1269d20afee32d4b/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/087fa46b30ee0c590f85d8cb0119608b36e56e0380571b37d67112d31fdc98c0/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d0af0fbcfa82766eae3a70df089ec9880016eac84f39e223e3f28e45ae44c862/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d5de87a8cd4302f0abd8373a5a706662eae7ae10283edec865b44399d9874c52/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/cb41b7f22ecfececd5e965f7abf3474d17aacefe71b3d837a9028a8f83840247/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d3e394544349a86dceaa96c717487a48a19d42416cd4b2496aab226a63cb8b61/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/0855b7b4660fda38fd44cfdf3cd102712dc40dd026e45d5a0c3d3d599ab3a393/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/706c8a2266735ffa43ad3e452e2aa82e26257b377d8c972c4913d0960c2176b3/rootfs
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/f9198705-cb41-48fe-a8a6-bc5b8377d704/volumes/kubernetes.io~secret/default-token-8thgc
tmpfs 1.9G 8.0K 1.9G 1% /var/lib/kubelet/pods/f8990ed2-a75e-45d6-9093-d13ae79729dc/volumes/kubernetes.io~secret/certificates
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/f8990ed2-a75e-45d6-9093-d13ae79729dc/volumes/kubernetes.io~secret/default-token-dcqr9
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/761179a6-6f11-4611-816b-2c1da9814b54/volumes/kubernetes.io~secret/default-token-8thgc
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/3d77d740d56a396211a9818630ec91e7e766327d122458a51a07c77525cb0809/rootfs
/dev/longhorn/pvc-73b89aea-3338-4155-83af-e2626fb650ec 9.8G 1.8G 8.0G 19% /var/lib/kubelet/pods/f8990ed2-a75e-45d6-9093-d13ae79729dc/volumes/kubernetes.io~csi/pvc-73b89aea-3338-4155-83af-e2626fb650ec/mount
shm 64M 12K 64M 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/158bae21bc1d83fe8c83316fc3edae0e81cd35fb295dcd06d4938c9957d947e8/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/158bae21bc1d83fe8c83316fc3edae0e81cd35fb295dcd06d4938c9957d947e8/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/b091e6e335a19a151df064c6bffe4a3a8c3b668c409dcb2703bd264371f5886d/rootfs
/dev/longhorn/pvc-77756bb4-21e4-4c05-a69c-c614e0ea1569 4.8G 544M 4.3G 12% /var/lib/kubelet/pods/761179a6-6f11-4611-816b-2c1da9814b54/volumes/kubernetes.io~csi/pvc-77756bb4-21e4-4c05-a69c-c614e0ea1569/mount
shm 64M 8.0K 64M 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/623eeecd47a036b6e06d9170a6a9c386d0afe4c00c353cee85ea0f1d55912880/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/623eeecd47a036b6e06d9170a6a9c386d0afe4c00c353cee85ea0f1d55912880/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/2e44472347fcf458cf71eddb6b52983dcf483bcd4748ebceb9c789d3a614a3e7/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/f2d4be789af94b35f67c2e1fbf6d5d3d082e6ee8915a11eff6ba4fbda5d5873c/rootfs
/dev/longhorn/pvc-cedad9ef-aca3-4e6e-8df3-e3f6559952ab 99G 52G 47G 53% /var/lib/kubelet/pods/f9198705-cb41-48fe-a8a6-bc5b8377d704/volumes/kubernetes.io~csi/pvc-cedad9ef-aca3-4e6e-8df3-e3f6559952ab/mount
shm 64M 12K 64M 1% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/e1ed31e97f4c7358dbf4352437a5aa14230bff0b40fff95b08812319b88738cb/shm
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e1ed31e97f4c7358dbf4352437a5aa14230bff0b40fff95b08812319b88738cb/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/87f869d31a718fe191476d540b37f67b6575115d43421eb7a3e62bfe6b4a90d8/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8d4dce4511018d9e1cb3f1cd0d9049de402d25d0342e07353d46f313f7309a30/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/94b4fd3fda664f68819d1fce9e8da0bacf4d8a31949402cb64575e138222e38e/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7074fedc3ebfad9ecd242fd5232b0edcab51154baec7cd3943ba64e3b8411e72/rootfs
overlay 25G 17G 6.9G 72% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/5cbca1ee8858615cf47b81c002d3ea758ce2c4616699d23609ea7b52e9cbb34e/rootfs
tmpfs 387M 0 387M 0% /run/user/0
And now the count is at: 3077853 After the reboot it is: 4363
I will did a reboot now because I don't want it to get unresponsive.
Maybe it helps that I have this only on a Debian host (Buster) and the master that runs on Ubuntu does not have this issue.
Debian runs on kernel: Linux s01-01 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux
Ubuntu runs on kernel: Linux s00 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
johansmitsnl
on 13 Nov 2019
Thanks for the info, I'm curious if this might be related to some version of systemd. If it continues to happen after the restart please share the output of:
lsof | sed 's/^.* //' | sort | uniq -c | sort -r -n | head
erikwilson
on 13 Nov 2019
@erikwilson
After reboot is is still an issue. Here the output:
$ lsof | sed 's/^.* //' | sort | uniq -c | sort -r -n | head
772985 [eventfd]
24628 directory)
7473 pipe
3070 (ESTABLISHED)
3004 type=STREAM
2101 [eventpoll]
1813 /
1758 TCP
1688 /dev/null
1264 UNIX
Is there any difference in the workloads running in Ubuntu or Debian? I suspect one of the pods may be the culprit, it looks like there is a lot of file related stuff running.
erikwilson
on 14 Nov 2019
Yes the workloads are different. How can I see which pod could be responsible for this?
I only started to notice this issue when switching to Debian and using the 0.10.0 with IPsec. Before it was running at Ubuntu as a single master with the same pods.
johansmitsnl
on 14 Nov 2019
Current output is:
$ lsof | sed 's/^.* //' | sort | uniq -c | sort -r -n | head
1713038 [eventfd]
24961 directory)
7652 pipe
3196 (ESTABLISHED)
2941 type=STREAM
2120 TCP
2099 [eventpoll]
1825 /
1692 /dev/null
1254 UNIX
Maybe this will help:
procs=$(ps -A -o pid,args)
openFiles=$(lsof)
pods=$(k3s crictl ps -a -o json | jq -r '.containers[] | .podSandboxId' | sort -u)
for pod in $pods; do
pid=$(echo "$procs" | grep $pod | awk '{ print $1 }')
[ "$pid" ] || continue
numOpen=$(echo "$openFiles" | grep -E "^container *$pid " | wc -l)
names=$(k3s crictl ps -a -o json | jq -r ".containers[] | select(.podSandboxId | match(\"$pod\")) | .metadata.name")
echo $numOpen $names
done | sort -r -n
@erikwilson thank you for the time but I'm running on the node but I end up with no count at all.
I ran it as a bash script:
0 traefik
0 tiller
0 replica-manager
0 qbittorrent sabnzbd jackett sonarr plex-media-server
0 longhorn-manager
0 longhorn-csi-plugin driver-registrar
0 local-path-provisioner local-path-provisioner
0 lb-port-8080 lb-port-443 lb-port-80
0 lb-port-51827
0 lb-port-32400
0 gitlab
0 engine-manager
0 engine-image-ei-3827e67c
0 database assistant
0 coredns
0 cert-manager cert-manager
0 cainjector cainjector
0 authentication-backend
0 authentication-backend
0 authentication-backend
0 authentication-backend
0 authentication-backend
0 authentication-backend
woops, ps format was off, updated the script
erikwilson
on 18 Nov 2019
Interesting:
241873 replica-manager
241854 engine-image-ei-3827e67c engine-image-ei-3827e67c
241779 engine-manager
1125 plex-media-server
765 lb-port-8080 lb-port-443 lb-port-80
585 longhorn-csi-plugin driver-registrar
546 database assistant
405 longhorn-manager longhorn-manager
378 lb-port-51827
378 gitlab-runner
378 gitlab
378 authentication-backend
378 authentication-backend
378 authentication-backend
378 authentication-backend
351 lb-port-32400
351 authentication-backend
351 authentication-backend
Looks like longhorn?
johansmitsnl
on 19 Nov 2019
Possibly, might be good to open an issue there with as much info as possible, and leave this one open just in case it is a weird system or dependency issue on our side. Thanks! :)
erikwilson
on 19 Nov 2019
Will do, thanks for the help so far!
johansmitsnl
on 19 Nov 2019
👍1
This is not related to Longhorn. Copy my comment from https://github.com/longhorn/longhorn/issues/901#issuecomment-558428455
This looks like related to Kubernetes(or k3s)'s implementation of probing. @erikwilson
The pods of abnormal large FDs are the pods with either livenessProbe or readinessProbe set. It's in fact interesting since the pods like engine-image-ei-3827e67c don't run anything except copy a file then wait for exiting, so it shouldn't use any more FDs after the copy is done.
And here is an example yaml for a pod of engine-image-ei-xxx
- args:
- -c
- cp /usr/local/bin/longhorn* /data/ && echo installed && trap 'rm /data/longhorn*
&& echo cleaned up' EXIT && sleep infinity
command:
- /bin/bash
image: longhornio/longhorn-engine:v0.7.0
imagePullPolicy: Always
name: engine-image-ei-ec95b5ad
readinessProbe:
exec:
command:
- ls
- /data/longhorn
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
I've tried another pod with livenessProbe set, the rate of lsof is increasing at the rate of about 4.5 FDs per second (with livenessProbe period set to 5 seconds). And lsof shows most of the FDs are [eventfd].
I've checked another RKE 1.14.6 setup, everything seems normal on that with the same Longhorn setup.
yasker
on 26 Nov 2019
I have been running the this as well and I have the same results. Pods which have probes set, use a ton more fd's than those that don't have them.
It doesn't seem to affect those with httpGet, only those with exec.
ambis
on 11 Dec 2019
The latest version of k3s (v1.0.1) still have same result even though it's latest docs mentioned Longhorn as Storage option. In my case, this issue causing unaccessible nodes due to exhausted fd resource
euiko
on 4 Jan 2020
This is not related to Longhorn. I'm not running it, and I have the exact same problem.
And culprit is obviously liveness/readiness probes with exec. At first, I had to reboot my servers once around three days. Then I took the liveness probes out of my own software. Now I get about a week before reboot. Still Calico, ingress-nginx, flux cd etc have their checks which eat up the fds.
ambis
on 4 Jan 2020
👍2
I believe this may be related to the embedded containerd. I noticed the exact same problem on Ubuntu 18.04.3 x64 and k3s 1.0.0 and after switching to Docker the problem went away.
jaredk-beta
on 6 Jan 2020
It's a blocker problem - issue needs labeling and prioritizing please
definitiontv
on 13 Jan 2020
👍2
Looks like this is being tracked with https://github.com/containerd/containerd/issues/3949 and there is a PR to fix at https://github.com/containerd/containerd/pull/3956
erikwilson
on 13 Jan 2020
👍4
Closing issue as it is resolved with v1.17.2-alpha3+k3s1.
ShylajaDevadiga
on 24 Jan 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
seanmalloy
路
3Comments
wpwoodjr
路
3Comments
giezi
路
3Comments
davidnuzik
路
3Comments
ashrafgt
路
3Comments
Most helpful comment
Looks like this is being tracked with https://github.com/containerd/containerd/issues/3949 and there is a PR to fix at https://github.com/containerd/containerd/pull/3956