K3d: [Help] Creating more than 30 nodes at once

[twobombs]

What did you do

For a couple of scaling tests I decided to spin up a k3d cluster with 100 agents

• How was the cluster created?

  • k3d cluster create amd24magny --servers 1 --agents 100

• What did you do afterwards?
  nothing

What did you expect to happen

creation of 100 agents

Concise description of what you expected to happen after doing what you described above.
Amount of containers spun up is 30 instead of ~102

Screenshots or terminal output

If applicable, add screenshots or terminal output (code block) to help explain your problem.

102 containers are created, 72 fail. output of log of one of the failed containers:

Key here is IMHO:

I1013 06:38:59.034310 6 kuberuntime_manager.go:217] Container runtime containerd initialized, version: v1.3.3-k3s2, apiVersion: v1alpha2
W1013 06:38:59.034630 6 probe.go:268] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
E1013 06:38:59.035071 6 plugins.go:598] Error initializing dynamic plugin prober: Error initializing watcher: too many open files
I1013 06:38:59.035398 6 server.go:1124] Started kubelet
I1013 06:38:59.035517 6 server.go:145] Starting to listen on 0.0.0.0:10250
E1013 06:38:59.036349 6 cri_stats_provider.go:375] Failed to get the info of the filesystem with mountpoint "/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data in memory cache.
E1013 06:38:59.036405 6 kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
E1013 06:38:59.037038 6 node.go:125] Failed to retrieve node info: nodes "k3d-amd24magny-agent-99" not found
I1013 06:38:59.071507 6 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
I1013 06:38:59.071629 6 volume_manager.go:265] Starting Kubelet Volume Manager
I1013 06:38:59.071734 6 desired_state_of_world_populator.go:139] Desired state populator starts to run
I1013 06:38:59.071874 6 server.go:393] Adding debug handlers to kubelet server.
E1013 06:38:59.087876 6 controller.go:228] failed to get node "k3d-amd24magny-agent-99" when trying to set owner ref to the node lease: nodes "k3d-amd24magny-agent-99" not found
E1013 06:38:59.115505 6 manager.go:244] Registration of the raw container factory failed: inotify_init: too many open files
F1013 06:38:59.115893 6 kubelet.go:1371] Failed to start cAdvisor inotify_init: too many open files

Which OS & Architecture

• Linux Ubuntu 20.04

Which version of k3d

Which version of docker

[iwilltry42]

Hi @twobombs and thanks for opening this issue!
Pretty cool load test you're doing there :)
However, I doubt that this is a bug in k3d itself, as, after all, k3d does not really care about the host's resources.
So either this is a bug in k3s or just a limit set by your host system.
The fatal error you outlined already is F1013 06:38:59.115893 6 kubelet.go:1371] Failed to start cAdvisor inotify_init: too many open files.
The inotify_limit is set by your system.
You can check the current limit with cat /proc/sys/fs/inotify/max_user_watches and temporarily increase it with sudo sysctl fs.inotify.max_user_watches=<whatever_suits_you> (For more info and the permanent solution, see e.g. this SO post: https://unix.stackexchange.com/a/13757)

[twobombs]

Tnx! Looked over that reference in the log.