Lokomotive: Self-hosted kubelet does not work on bare-metal, missing /etc/kubernetes/kubelet.config

[invidian]

I got this when trying to add Tinkerbell platform support for Lokomotive, based on bare-metal platform:

0 ✓ (106ms) 16:01:20 invidian@dellxps15mateusz ~/data/workspaces/tinkerbell $ klo kubelet-794qt
+ kubectl logs -f kubelet-794qt
./hyperkube kubelet --node-ip=192.168.50.10  --anonymous-auth=false --authentication-token-webhook --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ca.crt --cluster_dns=10.3.0.10 --cluster_domain=cluster.local --cni-conf-dir=/etc/kubernetes/cni/net.d --config=/etc/kubernetes/kubelet.config --kubeconfig=/etc/kubernetes/kubeconfig --lock-file=/var/run/lock/kubelet.lock --network-plugin=cni --pod-manifest-path=/etc/kubernetes/manifests --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --node-labels=node.kubernetes.io/master,node.kubernetes.io/controller=true --register-with-taints=node-role.kubernetes.io/master=:NoSchedule --address=192.168.50.10

Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
F0428 13:57:45.805795   19635 server.go:199] failed to load Kubelet config file /etc/kubernetes/kubelet.config, error failed to read kubelet config file "/etc/kubernetes/kubelet.config", error: open /etc/kubernetes/kubelet.config: no such file or directory

Looking at the source code, it seems we don't create kubelet.config file on bare-metal platform. We don't test self-hosted kubelet there either.

0 ✓ (68.0ms) 18:43:55 invidian@dellxps15mateusz ~/repos/kinvolk/lokomotive (invidian/tinkerbell-platform)*$ git grep kubelet.config
assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/cl/controller.yaml.tmpl:          --config=/etc/kubernetes/kubelet.config \
assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/cl/controller.yaml.tmpl:          cat <<EOF >/etc/kubernetes/kubelet.config
assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/cl/controller.yaml.tmpl:          apiVersion: kubelet.config.k8s.io/v1beta1
assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl:          --config=/etc/kubernetes/kubelet.config \
assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl:          cat <<EOF >/etc/kubernetes/kubelet.config
assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl:          apiVersion: kubelet.config.k8s.io/v1beta1
assets/lokomotive-kubernetes/bootkube/resources/charts/kubelet/templates/kubelet-ds.yaml:          --config=/etc/kubernetes/kubelet.config \
assets/lokomotive-kubernetes/kvm-libvirt/flatcar-linux/kubernetes/cl/controller.yaml.tmpl:          --config=/etc/kubernetes/kubelet.config \
assets/lokomotive-kubernetes/kvm-libvirt/flatcar-linux/kubernetes/cl/controller.yaml.tmpl:          cat <<EOF >/etc/kubernetes/kubelet.config
assets/lokomotive-kubernetes/kvm-libvirt/flatcar-linux/kubernetes/cl/controller.yaml.tmpl:          apiVersion: kubelet.config.k8s.io/v1beta1
assets/lokomotive-kubernetes/kvm-libvirt/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl:          --config=/etc/kubernetes/kubelet.config \
assets/lokomotive-kubernetes/kvm-libvirt/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl:          cat <<EOF >/etc/kubernetes/kubelet.config
assets/lokomotive-kubernetes/kvm-libvirt/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl:          apiVersion: kubelet.config.k8s.io/v1beta1
assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/cl/controller.yaml.tmpl:          --config=/etc/kubernetes/kubelet.config \
assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/cl/controller.yaml.tmpl:          cat <<EOF >/etc/kubernetes/kubelet.config
assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/cl/controller.yaml.tmpl:          apiVersion: kubelet.config.k8s.io/v1beta1
assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl:            --config=/etc/kubernetes/kubelet.config \
assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl:          cat <<EOF >/etc/kubernetes/kubelet.config
assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl:          apiVersion: kubelet.config.k8s.io/v1beta1
pkg/components/rook/manifests.go:  # kubelet directory path, if kubelet configured to use other than /var/lib/kubelet path.

[invidian]

We should minimize the following diff for both controllers and workers to resolve that:

git diff --no-index assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/cl/controller.yaml.tmpl assets/lokomotive-kubernetes/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml.tmpl
index 28e29677..ca8ad36c 100644
--- a/assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/assets/lokomotive-kubernetes/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml.tmpl
@@ -11,8 +11,8 @@ systemd:
             Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
             Environment="RKT_RUN_ARGS=--insecure-options=image"
             Environment="ETCD_NAME=${etcd_name}"
-            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
-            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
+            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
+            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
             Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
             Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
             Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
@@ -27,11 +27,19 @@ systemd:
             Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
             Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
             Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
-            ExecStopPost=-/opt/etcd-rejoin
     - name: docker.service
       enable: true
     - name: locksmithd.service
       mask: true
+    - name: kubelet.path
+      enable: true
+      contents: |
+        [Unit]
+        Description=Watch for kubeconfig
+        [Path]
+        PathExists=/etc/kubernetes/kubeconfig
+        [Install]
+        WantedBy=multi-user.target
     - name: wait-for-dns.service
       enable: true
       contents: |
@@ -47,13 +55,11 @@ systemd:
         RequiredBy=kubelet.service
         RequiredBy=etcd-member.service
     - name: kubelet.service
-      enable: true
       contents: |
         [Unit]
         Description=Kubelet via Hyperkube
         Wants=rpc-statd.service
         [Service]
-        ConditionPathExists=/etc/kubernetes/kubeconfig
         EnvironmentFile=/etc/kubernetes/kubelet.env
         Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
           --volume=resolv,kind=host,source=/etc/resolv.conf \
@@ -66,6 +72,10 @@ systemd:
           --mount volume=opt-cni-bin,target=/opt/cni/bin \
           --volume var-log,kind=host,source=/var/log \
           --mount volume=var-log,target=/var/log \
+          --volume iscsiconf,kind=host,source=/etc/iscsi/ \
+          --mount volume=iscsiconf,target=/etc/iscsi/ \
+          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
+          --mount volume=iscsiadm,target=/sbin/iscsiadm \
           --insecure-options=image"
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -76,7 +86,6 @@ systemd:
         ExecStartPre=/bin/mkdir -p /var/lib/calico
         ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
         ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=/etc/kubernetes/configure-kubelet-cgroup-driver
         ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --anonymous-auth=false \
@@ -86,8 +95,8 @@ systemd:
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --config=/etc/kubernetes/kubelet.config \
           --exit-on-lock-contention \
+          --hostname-override=${domain_name} \
           --kubeconfig=/etc/kubernetes/kubeconfig \
           --lock-file=/var/run/lock/kubelet.lock \
           --network-plugin=cni \
@@ -104,7 +113,7 @@ systemd:
     - name: bootkube.service
       contents: |
         [Unit]
-        Description=Bootstrap a Kubernetes cluster
+        Description=Bootstrap a Kubernetes control plane with a temp api-server
         ConditionPathExists=!/opt/bootkube/init_bootkube.done
         [Service]
         Type=oneshot
@@ -112,8 +121,6 @@ systemd:
         WorkingDirectory=/opt/bootkube
         ExecStart=/opt/bootkube/bootkube-start
         ExecStartPost=/bin/touch /opt/bootkube/init_bootkube.done
-        [Install]
-        WantedBy=multi-user.target
 storage:
   files:
     - path: /etc/kubernetes/kubelet.env
@@ -126,6 +133,12 @@ storage:
           KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
           NODE_LABELS="node.kubernetes.io/master,node.kubernetes.io/controller=true"
           NODE_TAINTS="node-role.kubernetes.io/master=:NoSchedule"
+    - path: /etc/hostname
+      filesystem: root
+      mode: 0644
+      contents:
+        inline:
+          ${domain_name}
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -158,68 +171,6 @@ storage:
             --net=host \
             --dns=host \
             --exec=/bootkube -- start --asset-dir=/assets "$@"
-    - path: /opt/etcd-rejoin
-      filesystem: root
-      mode: 0555
-      contents:
-        inline: |
-          #!/bin/bash
-          set -eou pipefail
-          # Rejoin a cluster as fresh node when etcd cannot join
-          # (e.g., after repovisioning, crashing or node being down).
-          # Set ExecStopPost=-/opt/etcd-rejoin to run when etcd failed and
-          # use env vars of etcd-member.service.
-          # Skip if not provisioned
-          if [ ! -d "/etc/ssl/etcd/" ]; then exit 0; fi
-          # or got stopped.
-          if [ "$EXIT_CODE" = "killed" ]; then exit 0; fi
-          now=$(date +%s)
-          if [ -f /var/lib/etcd-last-fail ]; then
-            last=$(cat /var/lib/etcd-last-fail)
-          else
-            last=0
-          fi
-          echo "$now" > /var/lib/etcd-last-fail
-          let "d = $now - $last"
-          # Skip and restart regularly if it does not fail within 120s.
-          if [ "$d" -gt 120 ]; then exit 0; fi
-          export ETCDCTL_API=3
-          urls=$(echo "$ETCD_INITIAL_CLUSTER" | tr "," "\n" | cut -d "=" -f 2 | tr "\n" "," | head -c -1)
-          # $$ for terraform
-          endpoints="$${urls//2380/2379}"
-          ARGS="--cacert=/etc/ssl/etcd/etcd-client-ca.crt --cert=/etc/ssl/etcd/etcd-client.crt --key=/etc/ssl/etcd/etcd-client.key --endpoints=$endpoints"
-          # Check if unhealthy (should be because etcd is not running)
-          unhealty=$((etcdctl endpoint health $ARGS 2> /dev/stdout | grep "is unhealthy" | grep "$ETCD_NAME") || true)
-          if [ -z "$unhealty" ]; then exit 0; fi
-          # Remove old ID if still exists
-          ID=$((etcdctl member list $ARGS | grep "$ETCD_NAME" | cut -d "," -f 1) || true)
-          if [ ! -z "$ID" ]; then
-            etcdctl member remove "$ID" $ARGS
-          fi
-          # Re-add as new member
-          etcdctl member add "$ETCD_NAME" --peer-urls="$ETCD_INITIAL_ADVERTISE_PEER_URLS" $ARGS
-          # Join fresh without state
-          mv /var/lib/etcd "/var/lib/etcd-bkp-$(date +%s)" || true
-          if [ -z "$(grep ETCD_INITIAL_CLUSTER_STATE=existing /etc/systemd/system/etcd-member.service.d/40-etcd-cluster.conf)" ]; then
-            echo 'Environment="ETCD_INITIAL_CLUSTER_STATE=existing"' >> /etc/systemd/system/etcd-member.service.d/40-etcd-cluster.conf
-            # Apply change
-            systemctl daemon-reload
-          fi
-          # Restart unit (yes, within itself)
-          systemctl restart etcd-member &
-    - path: /etc/kubernetes/configure-kubelet-cgroup-driver
-      filesystem: root
-      mode: 0744
-      contents:
-        inline: |
-          #!/bin/bash
-          set -e
-          readonly docker_cgroup_driver="$(docker info -f '{{.CgroupDriver}}')"
-          cat <<EOF >/etc/kubernetes/kubelet.config
-          apiVersion: kubelet.config.k8s.io/v1beta1
-          kind: KubeletConfiguration
-          cgroupDriver: "$${docker_cgroup_driver}"
-          EOF
 passwd:
   users:
     - name: core

[silvacraig]

plus one

[silvacraig]

Is there a branch with this patch applied that I could test? or some quick instructions on how to appply a patch - I'm unfamiliarr with the process

[invidian]

The draft PR addressing it, though it still require some thoughts: #436.

[silvacraig]

I ran with invidian/bare-metal-fix-self-hosted-kubelet and can report that it fixes the kubelet crash - running fine now