Kind: Kind control-plane not Ready since git OOM

[daixiang0]

What happened:
After kind create cluster --name test, wait 5m but control-plane node is still not ready.

Check with kubectl get nodes test-control-plane:

  Warning  SystemOOM                20s   kubelet, kind-control-plane     System OOM encountered, victim process: git, pid: 3171
  Warning  SystemOOM                20s   kubelet, kind-control-plane     System OOM encountered, victim process: git, pid: 3640
  Warning  SystemOOM                20s   kubelet, kind-control-plane     System OOM encountered, victim process: git, pid: 5500
  Warning  SystemOOM                20s   kubelet, kind-control-plane     System OOM encountered, victim process: git, pid: 5923
  Normal   NodeHasSufficientMemory  20s   kubelet, kind-control-plane     Node kind-control-plane status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    20s   kubelet, kind-control-plane     Node kind-control-plane status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     20s   kubelet, kind-control-plane     Node kind-control-plane status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  20s   kubelet, kind-control-plane     Updated Node Allocatable limit across pods
  Normal   Starting                 12s   kube-proxy, kind-control-plane  Starting kube-proxy.

What you expected to happen:
No OOM issue.

How to reproduce it (as minimally and precisely as possible):

kind create cluster --name test

Create-delete-Recreate and so on.

Anything else we need to know?:

I have deleted git from host, this issue still exit, i exec into node container and see the git OOM issue is inside the container indeed.

My host is a cloud VM and do not know the details about the underlying implementation.

Clean all containers and run docker system prune -f do no help, reboot also does no impact.

Environment:

• kind version: (use kind version): v0.7.0, also test v0.8.1
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Docker version: (use docker info):

Client:
 Debug Mode: false

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 114
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-91-generic
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.789GiB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

• OS (e.g. from /etc/os-release): 18.04.4 LTS (Bionic Beaver)

[tao12345666333]

can you show me your free's outputs?

[daixiang0]

$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           7.8G        421M        2.5G         11M        4.9G        7.1G
Swap:            0B          0B          0B

[daixiang0]

BTW, who call git? It is odd.

[BenTheElder]

kind does not call git

Do you have any workloads deployed ..?

[daixiang0]

Just start cluster, then kubectl describe nodes.

[BenTheElder]

How exactly did you confirm git was executed inside the container?

[daixiang0]

I have removed git binary at host. i told it in description.

[BenTheElder]

yes I read that ... but removing _a_ git binary on the host that doesn't mean that the kind container is invoking git ... you can find processes inside the container with e.g. docker exec kind-control-plane ps aux.

git is not installed in kind:
$ docker run --rm --entrypoint sh kindest/node:v1.18.2 -c 'git'
sh: 1: git: not found

[daixiang0]

Now it is flaky, i would confirm this issue in a week, if cannot reproduce, i wll close it.

[BenTheElder]

Any update?

[BenTheElder]

please reopen with more details if you have any in the future, this doesn't fit with anything kind does and isn't reproducible so far.