Kubeadm: How to update the configuration of a kubelet that is running a cluster

[vperson]

Didn't find the right answer after trying Google

After running the cluster with kubeadm for a while, the node node often fails to log in remotely due to the system hang.

Later, the default resource reservation for kubelet was found to be none. Now I want to make a kubelet configuration update for an already running cluster.

The following is the configuration of my initial cluster.

apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.0.1.166
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master-1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.13.3
networking:
  dnsDomain: cluster.local
  podSubnet: "10.244.0.0/16"
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
pod-infra-container-image: registry.aliyuncs.com/google_containers/pause:3.1
maxPods: 90
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs

After I updated the configuration.Got the following configuration

apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.0.1.166
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master-1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.13.3
networking:
  dnsDomain: cluster.local
  podSubnet: "10.244.0.0/16"
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
#pod-infra-container-image: registry.aliyuncs.com/google_containers/pause:3.1
#podInfraContainerImage: registry.aliyuncs.com/google_containers/pause:3.1
maxPods: 80
evictionHard:
  memory.available: "1024Mi"
  nodefs.available: "10%"
kubeReserved:
  cpu: "500m"
  memory: "1Gi"
systemReserved:
  cpu: "1"
  memory: "1Gi"
  ephemeral-storage: "10Gi"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs

Then executed the cluster configuration update command

kubeadm config upload from-file --config kubeadm-kubelet-update.yaml

Viewing the cluster configuration has not changed

$ kubeadm config view
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.13.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

$ kubectl get -n kube-system  configmaps kubelet-config-1.13 -o yaml
apiVersion: v1
data:
  kubelet: |
    address: 0.0.0.0
    apiVersion: kubelet.config.k8s.io/v1beta1
    authentication:
      anonymous:
        enabled: false
      webhook:
        cacheTTL: 2m0s
        enabled: true
      x509:
        clientCAFile: /etc/kubernetes/pki/ca.crt
    authorization:
      mode: Webhook
      webhook:
        cacheAuthorizedTTL: 5m0s
        cacheUnauthorizedTTL: 30s
    cgroupDriver: cgroupfs
    cgroupsPerQOS: true
    clusterDNS:
    - 10.96.0.10
    clusterDomain: cluster.local
    configMapAndSecretChangeDetectionStrategy: Watch
    containerLogMaxFiles: 5
    containerLogMaxSize: 10Mi
    contentType: application/vnd.kubernetes.protobuf
    cpuCFSQuota: true
    cpuCFSQuotaPeriod: 100ms
    cpuManagerPolicy: none
    cpuManagerReconcilePeriod: 10s
    enableControllerAttachDetach: true
    enableDebuggingHandlers: true
    enforceNodeAllocatable:
    - pods
    eventBurst: 10
    eventRecordQPS: 5
    evictionHard:
      imagefs.available: 15%
      memory.available: 100Mi
      nodefs.available: 10%
      nodefs.inodesFree: 5%
    evictionPressureTransitionPeriod: 5m0s
    failSwapOn: true
    fileCheckFrequency: 20s
    hairpinMode: promiscuous-bridge
    healthzBindAddress: 127.0.0.1
    healthzPort: 10248
    httpCheckFrequency: 20s
    imageGCHighThresholdPercent: 85
    imageGCLowThresholdPercent: 80
    imageMinimumGCAge: 2m0s
    iptablesDropBit: 15
    iptablesMasqueradeBit: 14
    kind: KubeletConfiguration
    kubeAPIBurst: 10
    kubeAPIQPS: 5
    makeIPTablesUtilChains: true
    maxOpenFiles: 1000000
    maxPods: 90
    nodeLeaseDurationSeconds: 40
    nodeStatusReportFrequency: 1m0s
    nodeStatusUpdateFrequency: 10s
    oomScoreAdj: -999
    podPidsLimit: -1
    port: 10250
    registryBurst: 10
    registryPullQPS: 5
    resolvConf: /etc/resolv.conf
    rotateCertificates: true
    runtimeRequestTimeout: 2m0s
    serializeImagePulls: true
    staticPodPath: /etc/kubernetes/manifests
    streamingConnectionIdleTimeout: 4h0m0s
    syncFrequency: 1m0s
    volumeStatsAggPeriod: 1m0s
kind: ConfigMap
metadata:
  creationTimestamp: "2019-02-18T09:45:25Z"
  name: kubelet-config-1.13
  namespace: kube-system
  resourceVersion: "153"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kubelet-config-1.13
  uid: eb08bbd9-3361-11e9-8888-000c299021ee

How can I use kubeadm to update the configuration of a kubernetes cluster that is already running?

[neolit123]

/triage support

Viewing the cluster configuration has not changed

the kubeadm config command is what you can use to modify the kubeadm CluserConfiguration.

Didn't find the right answer after trying Google

here are the official docs for this. you need the DynamicKubeletConfiguration feature:
https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/

also see this:
https://stackoverflow.com/questions/49147294/kubelet-dynamic-configuration-doesnt-work

kubeadm also has some commands for that, but the feature is experimental for the time being see kubeadm alpha kubelet config --help.

i personally couldn't get this to work as expected.

[vperson]

thanks you

[eskuai]

i wanna upgrade all kubelet the following options
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serializeImagePulls: true
taintBasedEvictions=true
--image-pull-progress-deadline 5m0s duration

how can i change and upgrade?
i cannot get nothing about --image-pull-progress-deadline

[jessehu]

the kubeadm config command is what you can use to modify the kubeadm CluserConfiguration.

Hi @neolit123, I didn't find the kubeadm config sub command to modify the kubeadm CluserConfiguration. Could you please share more details? See https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/.
I'd like to update some configuration for the K8s components (e.g apiserver, scheduler) in a K8s cluster created via kubeadm init --extraArgs=xxx=yyy. Is there a command like kubeadm apply --extraArgs=xxx=zzz to update the configuration? Thanks!

[neolit123]

@eskuai

updating the running kubelet is only possibly using the feature DynamicKubeletConfig
https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/

however it's not enabled by default so if you created the cluster without it you cannot use it.

alternatively you can do the following steps:

• using kubectl edit modify the kubelet-config-x.yy ConfigMap in the cluster.

then you have two options:

1) on all nodes open the file /var/lib/kubelet/config.yaml make your modifications and restart the kubelet using sudo systemctl restart kubelet.

or

2) modify the above yaml file only on a control-plane node, restart the kubelet, wait for the kubelet and api-server to come up and then rejoin the rest of the nodes.

better UX can possibly come in the future:
https://github.com/kubernetes/kubeadm/issues/970

[neolit123]

@jessehu

I'd like to update some configuration for the K8s components (e.g apiserver, scheduler) in a K8s cluster created via kubeadm init --extraArgs=xxx=yyy. Is there a command like kubeadm apply --extraArgs=xxx=zzz to update the configuration? Thanks!

updating parameters of the control-plane components is currently not supported.

users are using kubeadm upgrade apply [version] --config update-config.yaml, however this is not recommended. use at your own risk!!

the alternative is to (also use at your own risk!):

• edit the kubeadm-config ConfigMap using kubectl edit.

then you have two options:

1) go to all nodes and modify /etc/kubernetes/manifests/*.yaml

or

2) modify /etc/kubernetes/manifests/*.yaml only on a control-plane node, then rejoin the rest of the control-plane nodes.

please also track:
https://github.com/kubernetes/kubeadm/issues/970

[jessehu]

Thanks a lot @neolit123. I ran kubeadm config view > kubeadm-config.yaml and added a new configuration for apiserver in kubeadm-config.yaml(e.g. enable PSP), then ran kubeadm upgrade apply --config kubeadm-config.yaml to update the k8s cluser with new configuration. It works.
Manually updating the configuration file should work, but seems not user friendly.
I found that minikube supports reconfiguring the existing k8s cluster via minikube CLI options, e.g. minikube start ... --extra-config apiserver.enable-admission-plugins=PodSecurityPolicy just same as using minikube start ... to create the k8s. It would be great kubeadm officially supports this kind of k8s cluster reconfiguration.

[neolit123]

yes, but minikube is single node.

for kubeadm please track:
https://github.com/kubernetes/kubeadm/issues/970
(we moved it to a new ticket).

[jessehu]

Thanks @neolit123 for pointing out the issue #970. Since kubeadm init and kubeadm join ran on a single node separately, I think if minikube can do it (on a single node), kubeadm can do it too.

[alexcpn]

/triage support

Viewing the cluster configuration has not changed

the kubeadm config command is what you can use to modify the kubeadm CluserConfiguration.

Didn't find the right answer after trying Google

here are the official docs for this. you need the DynamicKubeletConfiguration feature:
https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/

also see this:
https://stackoverflow.com/questions/49147294/kubelet-dynamic-configuration-doesnt-work

kubeadm also has some commands for that, but the feature is experimental for the time being see kubeadm alpha kubelet config --help.

i personally couldn't get this to work as expected.

kubeadm alpha kubelet config enable-dynamic --node-name azuretest-2 --kubelet-version 1.17.0
[kubelet] Enabling Dynamic Kubelet Config for Node "azuretest-2"; config sourced from ConfigMap "kubelet-config-1.17" in namespace kube-system
[kubelet] WARNING: The Dynamic Kubelet Config feature is beta, but off by default. It hasn't been well-tested yet at this stage, use with caution.

However, this does not work; Even after setting this and following

What worked

https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
In all worker nodes add the config you want 
[root@azuretest-2 ~]# cat /var/lib/kubelet/kubeadm-flags.env 
KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --container-runtime=remote --container-runtime-endpoint=/var/run/containerd/containerd.sock --resolv-conf=/etc/resolv.conf --max-pods=700"
systmectl daemon-reload
systemctl restart kubelet