Cluster-api-provider-azure: NMI fails to run on hosts which have no /etc/default/kubelet

[invidian]

/kind bug

What steps did you take and what happened:

1. Setup CAPZ with Tilt for development on cluster with no /etc/default/kubelet file.
2. Tilt never converges with NMI pod in crash loopback with following logs:

$ klo capz-nmi-lc4g7
+ kubectl logs -f capz-nmi-lc4g7
F0312 16:12:33.667869       1 main.go:78] failed to check if CNI plugin is kubenet, error: read /etc/default/kubelet: is a directory
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc0000a6001, 0xc0000ce0e0, 0x85, 0xda)
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:996 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x2188a40, 0xc000000003, 0x0, 0x0, 0xc00044b960, 0x20ebf7e, 0x7, 0x4e, 0x0)
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:945 +0x191
k8s.io/klog/v2.(*loggingT).printf(0x2188a40, 0x3, 0x0, 0x0, 0x16cce4a, 0x34, 0xc0005bfea0, 0x1, 0x1)
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:733 +0x17a
k8s.io/klog/v2.Fatalf(...)
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1463
main.main()
        /go/src/github.com/Azure/aad-pod-identity/cmd/nmi/main.go:78 +0x29b
goroutine 5 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x2188a40)
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1131 +0x8b
created by k8s.io/klog/v2.init.0
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:416 +0xd8
goroutine 6 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00003a140)
        /go/pkg/mod/[email protected]/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
        /go/pkg/mod/[email protected]/stats/view/worker.go:32 +0x57

What did you expect to happen:

NMI converges, or ideally, NMI is not deployed when deploying to management cluster not running on Azure

Anything else you would like to add:

Workaround is to run sudo rm -r /etc/default/kubelet/ && sudo touch /etc/default/kubelet on your worker nodes.

I think such patch should fix this:

diff --git config/default/aad-pod-identity-deployment.yaml config/default/aad-pod-identity-deployment.yaml
index 50addc19..bf4e453f 100644
--- config/default/aad-pod-identity-deployment.yaml
+++ config/default/aad-pod-identity-deployment.yaml
@@ -103,6 +103,7 @@ spec:
       - name: kubelet-config
         hostPath:
           path: /etc/default/kubelet
+          type: FileOrCreate
       containers:
       - name: nmi
         image: "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.1"

Environment:

• cluster-api-provider-azure version: 02ca6c383c4c459efb79c8d3673f332f212f4046
• Kubernetes version: (use kubectl version): Deployed with Lokomotive 0.6.0+git.

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"archive", BuildDate:"2021-02-19T08:41:57Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:09:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):

NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=2605.12.0
VERSION_ID=2605.12.0
BUILD_ID=2021-01-27-1731
PRETTY_NAME="Flatcar Container Linux by Kinvolk 2605.12.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"

[CecileRobertMichon]

@invidian is it specific to the flatcar image you're using that /etc/default/kubelet doesn't exist?

@aramase @chewong does NMI / pod identity have an explicit requirement on the existence of /etc/default/kubelet?

[invidian]

@invidian is it specific to the flatcar image you're using that /etc/default/kubelet doesn't exist?

I guess it entirely depends on how you deploy K8s for management cluster. If you run everything in containers, you won't have this file either.

[aramase]

@aramase @chewong does NMI / pod identity have an explicit requirement on the existence of /etc/default/kubelet?

NMI uses /etc/default/kubelet to determine if the CNI plugin is kubenet. We're adding an option in the next release to skip checking the file if --allow-network-plugin-kubenet is set to true.

[CecileRobertMichon]

@aramase thanks. Does the workaround suggested by @invidian above seem appropriate to you? kubeadm / CAPZ doesn't support kubenet.

Workaround is to run sudo rm -r /etc/default/kubelet/ && sudo touch /etc/default/kubelet on your worker nodes.

I think such patch should fix this:

diff --git config/default/aad-pod-identity-deployment.yaml config/default/aad-pod-identity-deployment.yaml
index 50addc19..bf4e453f 100644
--- config/default/aad-pod-identity-deployment.yaml
+++ config/default/aad-pod-identity-deployment.yaml
@@ -103,6 +103,7 @@ spec:
       - name: kubelet-config
         hostPath:
           path: /etc/default/kubelet
+          type: FileOrCreate
       containers:
       - name: nmi
         image: "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.1"

[aramase]

@aramase thanks. Does the workaround suggested by @invidian above seem appropriate to you? kubeadm / CAPZ doesn't support kubenet.

Yeah, that's a valid fix. We'll also add this to our manifests as part of the next release.

[aramase]

Opened https://github.com/Azure/aad-pod-identity/issues/1002 to track the change

[CecileRobertMichon]

Thanks! In the meantime, I'll mark this as help wanted to add it to the CAPZ NMI spec.

/help

[k8s-ci-robot]

@CecileRobertMichon:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Thanks! In the meantime, I'll mark this as help wanted to add it to the CAPZ NMI spec.

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nader-ziada]

/assign