Kubeadm: KubeletExtraArgs not being read from config file

[Jaskaranbir]

What keywords did you search in kubeadm issues before filing this one?

args (such extensive search, much results)

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version): v1.11.1

Environment:

• Kubernetes version (use kubectl version): v1.11.1
• Cloud provider or hardware configuration: Local VM in Vagrant
• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Kernel (e.g. uname -a): Linux k8s-m1 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

Kubeadm doesn't consider KubeletExtraArgs. The effect of the command is not applied (Kubelet doesn't run with specified args).

What you expected to happen?

Kubelet should have run with specified args.

How to reproduce it (as minimally and precisely as possible)?

Init Kubeadm with config file:

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
kubernetesVersion: ...

api:
  advertiseAddress: ...

networking:
  dnsDomain: ...
  serviceSubnet: ...
  podSubnet: ...

bootstrapToken:
  token: ...

nodeRegistrationOptions:
  taints: [] # Single-node cluster
  kubeletExtraArgs:
    fail-swap-on: "false" # Doesn't apply. Kubelet still crashes with "swap-on" error.

Anything else we need to know?

How to confirm the issue? Check the contents of /etc/default/kubelet. It should have had the fail-swap-on arg (afaik), but it doesn't have any arguments (none).

Also, everything works fine when the args are added manually to /etc/default/kubelet. So if I change file-contents...

From (default generated):
KUBELET_EXTRA_ARGS=
To:
KUBELET_EXTRA_ARGS=--fail-swap-on=false

...everything works as intended, and kubelet runs with specified args when we kubeadm init again.

Most likely the issue is probably somewhere in https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/phases/kubelet.go (from some initial exploration), but I am too busy at the moment to precisely track it.

[neolit123]

hi, @Jaskaranbir

please have a look at our example config for 1.11.
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
reference docs:
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#MasterConfiguration

in your example, this should be changed:

apiVersion: kubeadm.k8s.io/v1alpha1

to:

apiVersion: kubeadm.k8s.io/v1alpha2

in 1.11 kubeadm uses the v1alpha2 type, not v1alpha1.

this:

bootstrapToken:
  token: ...

should look like:

bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication

also:
fail-swap-on: "false"
should be:
fail-swap-on: false

please, respond if this fixes your kubelet extra args problem.

[Jaskaranbir]

Hi @neolit123
Thanks for the reply. However, my issue is still not resolved (Kubelet still doesn't get the args).

Here's my new config file:

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: ...

api:
  advertiseAddress: ...

networking:
  dnsDomain: ...
  serviceSubnet: ...
  podSubnet: ...

bootstrapToken:
  - groups:
      - system:bootstrappers:kubeadm:default-node-token
    token: ...
    ttl: 24h0m0s
    usages:
      - signing
      - authentication

nodeRegistrationOptions:
  taints: []
  kubeletExtraArgs:
    fail-swap-on: false

I am still facing the swap-error when starting Kubelet, and /etc/default/kubelet still doesn't have any args.

[Jaskaranbir]

EDIT (for my last comment):

Thanks for help; I just realized that I was using wrong options to configure. This is what worked for me:

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: ...

api:
  advertiseAddress: ...

networking:
  dnsDomain: ...
  serviceSubnet: ...
  podSubnet: ...

bootstrapToken:
  - groups:
      - system:bootstrappers:kubeadm:default-node-token
    token: ...
    ttl: 24h0m0s
    usages:
      - signing
      - authentication

kubeletConfiguration:
  baseConfig:
    cgroupDriver: cgroupfs
    cgroupsPerQOS: true
    failSwapOn: false

nodeRegistrationOptions:
  taints: []

Although I am still not sure what the KubeletExtraArgs is for then (https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1alpha2/types.go#L147).

In any case, closing this for now. Thanks again for the help.

[luxas]

nodeRegistrationOptions:
  taints: []

should be

nodeRegistration:
  taints: []
  kubeletExtraArgs:
    foo: bar

As can be seen here: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1alpha2/types.go#L42

[Jaskaranbir]

@luxas thanks for the reply, but I already tried that earlier:

nodeRegistrationOptions:
  taints: []
  kubeletExtraArgs:
    fail-swap-on: false

However, the Kubelet still failed with the swap error. Am I missing something here?

Another thing that confused me a bit is the taints key.
If its a blank array (as in my example), the node should not have any taints, correct (haven't really played around with YAML parsing in Go, so have a little idea of how slices map to YAML arrays)?
However, my master node still gets the NoSchedule traits and I have to manually remove it again after the setup. Not a big deal, but would help if I could remove the extra last step.

[luxas]

It needs to be nodeRegistration not nodeRegistrationOptions

[Jaskaranbir]

Ah.... guess I should take some break lol
Thanks for the help.