Kubeadm: apiserver fails to start because livenessprobe is too aggressive

[anguslees]

[Lubomir] NOTE: possible fix was submitted here:
https://github.com/kubernetes/kubernetes/pull/66264

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3+2c2fe6e8278a5", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"not a git tree", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/arm"}

Environment:

• Kubernetes version (use kubectl version):
  Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:30:51Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/arm"}

• Cloud provider or hardware configuration:
  arm32 (bananapi - basically a raspberrypi2)

• OS (e.g. from /etc/os-release):
  (my own OS image)
  ID="containos"
  NAME="containos"
  VERSION="v2017.07"
  VERSION_ID="v2017.07"
  PRETTY_NAME="containos v2017.07"

• Kernel (e.g. uname -a):
  Linux master2 4.9.20 #2 SMP Wed Aug 16 15:36:20 AEST 2017 armv7l GNU/Linux

• Others:

What happened?

kubeadm init sits ~forever at the "waiting for control plane" stage. docker ps/logs investigation shows apiserver is being killed (SIGTERM) and restarted continuously.

What you expected to happen?

Everything to work :) In particular, apiserver to come up and the rest of the process to proceed.

How to reproduce it (as minimally and precisely as possible)?

Run kubeadm init on a slow machine.

Anything else we need to know?

For me, during the churn of all those containers starting at once, it takes apiserver about 90s(!) from its first log line to responding to HTTP queries. I haven't looked in detail at what it's doing at that point, but the logs mention what looks like etcd bootstrapping things.

My suggested fix is to set apiserver initialDelaySeconds to 180s. And probably similar elsewhere in general - I think there's very little reason to have aggressive initial delays.

(Unless you're a unittest that expects to frequently encounter failures, my experience with production software suggests the correct solution to timeouts is almost always to have waited longer).

[pipejakob]

It seems that we set both InitialDelaySeconds and TimeoutSeconds to 15 for the control plane pods currently, which matches what kube-up.sh does as well. I understand the initial launch being slow, with all of the images are being pulled at once, but after all of the images have been pulled, how long does it take your apiserver to respond to /healthz checks after being launched?

No doubt we should probably tweak both of these values to accommodate lower-power machines.

[anguslees]

Once started it can respond to health checks in << 15s - it's really just all the extra stuff that the apiserver does between exec() and actually being ready to serve afaics.

[anguslees]

oh, and the docker pull time doesn't count against InitialDelaySeconds afaics (good). In other examples with larger (generic ubuntu) images over my slow network link, the pull can take many minutes but the initialDelaySeconds timer doesn't seem to start ticking until the pull has completed and the docker run has started. (I haven't looked at the relevant code - just frequent anecdotal experience)

[koalalorenzo]

I am running the same problem. With slow machines kubeadm sits forever. Using v1.7.4

[pipejakob]

@anguslees and @koalalorenzo, can you confirm that if you manually change the liveness probe settings (by editing the manifest files in /etc/kubernetes/manifests/) that this fixes your problem? I also recently saw a case on Slack where the user had the same symptoms but was likely running into memory constraints, because the problem went away when they moved to a host type with more memory.

I just want to make sure that this approach will actually fix the problem before we invest time into coding it up. Thanks!

[praseodym]

I'm also experiencing this issue when attempting to use kubeadm in QEMU without hardware-assisted virtualisation (which is a bad idea because it is terribly slow). Increasing the InitialDelaySeconds and TimeoutSeconds helps; the cluster will then eventually come up.

[anguslees]

@pipejakob I can confirm that (on my bananapi) running this in another terminal at the right point in the kubeadm run makes everything come up successfully:

sed -i 's/initialDelaySeconds: [0-9]\+/initialDelaySeconds: 180/' /etc/kubernetes/manifests/kube-apiserver.yaml

(I usually also manually docker kill the old/restart-looping apiserver container, I'm not sure if that gets cleaned up automatically with static pods)

[pipejakob]

@anguslees Great! Thanks for confirmation.

[alexjmoore]

I can confirm I've also just had this issue, in my case on an raspberry pi 3. Changing to 180s fixed it, however I think I also encountered issue #106 as in my case it simply died with:

Sep 1 10:47:30 raspberrypi kubelet[6053]: W0901 10:47:30.020409 6053 kubelet.go:1596] Deleting mirror pod "kube-apiserver-raspberrypi_kube-system(7c03df63-8efa-1
1e7-ae86-b827ebdd4b52)" because it is outdated

I had to manually HUP the kubelet process to get it to kick back to life.

[Tha-Fox]

I can also confirm I had this and wanted to say thank you for saving my sanity. I have Raspberry Pi 2B and I was stuck at init phase for the last month. After running that one-liner once it started waiting for the control plane, I got it going forward.

[anguslees]

This issue still exists in kubeadm v1.8.0, and it's worse because kubeadm itself now has a 1min timeout on most actions. The 1min timeout seems to have been chosen arbitrarily, and unfortunately a) doesn't give you enough time to jump in and debug/workaround the issue (eg: sed hack above), b) enough time for the apiserver to start up (~90s for me), even when initialDelaySeconds has been extended, and c) can't be increased without hacking/rebuilding kubeadm afaics.

Timeouts break otherwise correct logic, particularly in complex eventually-consistent systems - we should never add them "just because" :( My understanding is that kubeadm is meant to be a building block on which larger deployment systems can depend. Consequently I boldly propose removing all timeouts from kubeadm itself (the various phases should continue retrying forever), and rely on the higher level process to add an overall timeout if/when appropriate in that higher level context. In the simple/direct use case this would mean "retry until user gives up and presses ^c". Would such a PR be acceptable?

[luxas]

@anguslees We had the "wait forever" behavior earlier; but that was very sub-optimal from an UX PoV, so now we do have timeouts. We might want to increase some of those timeouts if you want.

The problem is that usage of kubeadm is two-fold. We both have users typing kubeadm interactively that want to know if something is happening or not _and_ higher-level consumers.

[anguslees]

.. So what direction are we going to go here? Currently I use a fork of kubeadm that has numerous timeouts cranked up 10x, and I'd _like_ to believe that I can go back to using standard kubeadm binaries at some point.

[kargakis]

@anguslees We had the "wait forever" behavior earlier; but that was very sub-optimal from an UX PoV, so now we do have timeouts. We might want to increase some of those timeouts if you want.

How about making them configurable? Does it make sense to have a single option that owns all of them?

[kargakis]

/priority important-soon

[luxas]

Does it make sense to have a single option that owns all of them?

Probably, or some kind of "weight" for all the timeouts to be multiplied with... Otherwise we'll get into config hell with 20 different types of timeout flags :)

[rsterrenburg]

Running into the same issue using kubeadm upgrade on raspberry pi 2 cluster. Upgrade fails due to agressive timeouts. Changing the liveness probe settings in the manifests doesn't help. Any ideas?

[anguslees]

I still propose a pattern where any kubeadm timeout is inherited from the calling context (or part of a more sophisticated error recovery strategy), rather than sprinkling arbitrary timeouts throughout the lower levels of the kubeadm codebase.

In its simplest form, this would behave almost exactly like removing all the timeouts from kubeadm and replacing them with one overall "run for xx mins, then abort if not finished" global timer (since kubeadm can't do much in the way of error recovery other than just waiting longer).

For the original manifest livenessProbe timeouts, it's literally a one-liner patch. Unfortunately, fixing the livenessProbe alone is no longer sufficient since the "deviation from normal == error" fallacy has spread further throughout the kubeadm codebase. Changing cultural awareness is hard, so in the meantime I have a forked version of kubeadm here, if anyone wants to just install onto a raspberry pi. (Build with make cross WHAT=cmd/kubeadm KUBE_BUILD_PLATFORMS=linux/arm)

[dvdmuckle]

@anguslees Do you have a compiled 1.9.4 version of your patched kubeadm? I'm having trouble compiling your patched version.

I'm surprised kubeadm doesn't have this behavior behind a flag. Perhaps a PR is in order?

[timothysc]

/assign @liztio

[timothysc]

So we've fixed two issues in 1.11 which could affect this.

1. Prepull the etcd image prior to startup.
2. Fix the race-condition check on startup.
   ...

If this is only happening on rasberry pi - gear then we need some way of qualifying a lowest common denominator.

What is the lowest target platform that we are planning to support?

[praseodym]

I think Raspberry Pi 2 is the (s)lowest platform you'd want to run Kubernetes on. My tests with non-hardware assisted QEMU are too exotic to be taken into account.

Considering that the Pi suffers from slow I/O a lot, pre-pulling all images will already help a lot, but we will need some real-world tests to determine the actual timeouts.

[timothysc]

imo rasberry pi2 is too old to support - http://socialcompare.com/en/comparison/raspberrypi-models-comparison, came out in 2015.

>= 3 seems more reasonable imo.

[anguslees]

Prepulling images won't help. The livenessProbe timer doesn't start until after the image is pulled (as I pointed out above).

The fix is just to extend the initialDelaySeconds timeout(s). The current timeout values in kubeadm are being misused to "enforce" a fast UX experience, and not error detection.

Edit: And to be clear, it's only the startup that takes ages - my control cluster operates on 3xRPi2 just fine, once I increase the apiserver intialDelaySeconds timeout (and other install-only timeouts used within kubeadm itself). I don't understand why we're still talking about this :/

[carlosedp]

I have an ARM64 cluster on 1.9.3 and successfully updated to 1.9.7 but got the timeout problem to upgrade from 1.9.7 to 1.10.2.

I even tried editing and recompiling kubeadm increasing the timeouts (like these last commits https://github.com/anguslees/kubernetes/commits/kubeadm-gusfork) with same results.

$ sudo kubeadm upgrade apply  v1.10.2 --force
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/version] You have chosen to change the cluster version to "v1.10.2"
[upgrade/versions] Cluster version: v1.9.7
[upgrade/versions] kubeadm version: v1.10.2-dirty
[upgrade/version] Found 1 potential version compatibility errors but skipping since the --force flag is set:

   - Specified version to upgrade to "v1.10.2" is higher than the kubeadm version "v1.10.2-dirty". Upgrade kubeadm first using the tool you used to install kubeadm
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.10.2"...
Static pod: kube-apiserver-kubemaster1 hash: ed7578d5bf9314188dca798386bcfb0e
Static pod: kube-controller-manager-kubemaster1 hash: e0c3f578f1c547dcf9996e1d3390c10c
Static pod: kube-scheduler-kubemaster1 hash: 52e767858f52ac4aba448b1a113884ee
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-kubemaster1 hash: 413224efa82e36533ce93e30bd18e3a8
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests346927148/etcd.yaml"
[certificates] Using the existing etcd/ca certificate and key.
[certificates] Using the existing etcd/server certificate and key.
[certificates] Using the existing etcd/peer certificate and key.
[certificates] Using the existing etcd/healthcheck-client certificate and key.
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests190581659/etcd.yaml"
[upgrade/staticpods] Not waiting for pod-hash change for component "etcd"
[upgrade/etcd] Waiting for etcd to become available
[util/etcd] Waiting 30s for initial delay
[util/etcd] Attempting to get etcd status 1/10
[util/etcd] Attempt failed with error: dial tcp 127.0.0.1:2379: getsockopt: connection refused
[util/etcd] Waiting 15s until next retry
[util/etcd] Attempting to get etcd status 2/10
[util/etcd] Attempt failed with error: dial tcp 127.0.0.1:2379: getsockopt: connection refused
[util/etcd] Waiting 15s until next retry
[util/etcd] Attempting to get etcd status 3/10
[util/etcd] Attempt failed with error: dial tcp 127.0.0.1:2379: getsockopt: connection refused
[util/etcd] Waiting 15s until next retry
[util/etcd] Attempting to get etcd status 4/10
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests346927148"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests346927148/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests346927148/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests346927148/kube-scheduler.yaml"
[upgrade/staticpods] The etcd manifest will be restored if component "kube-apiserver" fails to upgrade
[certificates] Using the existing etcd/ca certificate and key.
[certificates] Using the existing apiserver-etcd-client certificate and key.
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests190581659/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: [timed out waiting for the condition]

[ghoben]

In case kubeadm is used and the apiserver is started up, we can try to measure points on first startup. Maybe we alter the configuration in a later stage for the timeouts adapted on the measurements on first initialization. Also it is hard to find out, that the apiserver is kicked by the healtz check looking at the logs, we may at least get a better logging in place to be aware of the problem. It took me quite sometime to find out that the livenessprobe was the problem. I have to mention I'm a beginner, and that would at least be helpful to be mentioned somewhere on the failure output for kubeadm.

[brendandburns]

RaspberryPi 3 still demonstrates this issue, even with pre-pulled images. The API server takes 2-3 minutes to get to a place where it can serve a page...

It would be great to have this be configurable, because right now, I'm monkey patching the YAML file in the background while kubeadm runs.

[brendandburns]

@carlosedp what I do during an upgrade is take down the kubelet while the apiserver is booting. It's kind of hacky, but it prevents the health-check from firing until the apiserver is up.

But honestly kubeadm upgrade and ARM just don't work that well together in my experience...

[carlosedp]

@brendandburns It worked perfectly until 1.9. I deployed my 1.8 cluster with it with no hiccups, then upgraded to 1.9 two times. No idea on what might have changed in 1.10 to cause this problems.

I saw that the timeout changed in 1.11 to 5 minutes (https://github.com/kubernetes/kubernetes/pull/64988/files#diff-2056df5f0c3a4828b3f9a2510a7533c7L45). Have you tried with 1.11?

Gonna try this hack after I return from vacations. Thanks for the tip!

[neolit123]

@brendandburns @carlosedp
yes, please try 1.11 to confirm that the timeout increase is a fix for you.

/cc @kubernetes/sig-cluster-lifecycle-bugs

[DJGummikuh]

Hey! I'm also encountering this issue.
Interestingly though, I manage to build a cluster master from scratch on my Raspberry 3, but consistenly fail to on my 3+.
Anyways, the Version I'm currently using (as per the step-by-step documentation at https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/ ) is
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/arm"}

As with the others, the apiserver container does get up eventually, but not before kubeadm bails out, leaving me in limbo as I'm too unexperienced to manually pick up from there.

Quick Update: running
watch -n 1.0 "sed -i 's/initialDelaySeconds: [0-9]\+/initialDelaySeconds: 180/' /etc/kubernetes/manifests/kube-apiserver.yaml"
in a separate terminal allowed my cluster to get up.

[neolit123]

@DJGummikuh
thanks for testing 1.11.

As with the others, the apiserver container does get up eventually, but not before kubeadm bails out, leaving me in limbo as I'm too unexperienced to manually pick up from there.

how much time does it take for the apiserver to start in your case?
it seems that we may have to make this configurable.

[DJGummikuh]

It's hard to say, I'd guess approx 1 minute, however I don't know how to properly measure that.

Additionally, now that my master is operational, I fail adding a node to it with what seems to be another timeout issue.
`[preflight] running pre-flight checks
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_rr ip_vs_wrr ip_vs_sh ip_vs] or no builtin kernel ipvs support: map[ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{}]
you can solve this problem with following methods:

1. Run 'modprobe -- ' to load missing kernel modules;
   
   
   
   1. Provide the missing builtin kernel ipvs support
   
   

I0708 19:02:20.256325 8667 kernel_validator.go:81] Validating kernel version
I0708 19:02:20.256846 8667 kernel_validator.go:96] Validating kernel config
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.03.1-ce. Max validated version: 17.03
[discovery] Trying to connect to API Server "192.168.2.2:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.2.2:6443"
[discovery] Requesting info from "https://192.168.2.2:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.2.2:6443"
[discovery] Successfully established connection with API Server "192.168.2.2:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] It seems like the kubelet isn't running or healthy.
Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
timed out waiting for the condition`

During this time, not a single docker container shows up on my node.

[neolit123]

[WARNING RequiredIPVSKernelModulesAvailable]:

offtopic, we are talking about this here:
https://github.com/kubernetes/kubeadm/issues/975

Additionally, now that my master is operational, I fail adding a node to it with what seems to be another timeout issue.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.

the kubelet could not start. better look at the kubelet logs.

- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'

the liveness probe should be made configurable, yet we should probably talk about the best way of doing that in the kubeadm config.

i think these are the values used, so if you are building kubeadm yourself try playing with:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/staticpod/utils.go#L95-L96
but mind that this would increase the timeouts on all control plane components.

[DJGummikuh]

Edit: I'm apparently too stupid to properly format comments in Github :-(

Here are the log outputs of both systemctl status kubelet and journalctl -xeu kubelet. "No cloud provider specified is the only thing that springs to eye.

`root@djg-clusterpi3:/home/djgummikuh# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Sun 2018-07-08 19:55:15 CEST; 2min 12s ago
     Docs: http://kubernetes.io/docs/
 Main PID: 9233 (kubelet)
   Memory: 14.4M
      CPU: 17.064s
   CGroup: /system.slice/kubelet.service
           └─9233 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-pl

Jul 08 19:55:15 djg-clusterpi3 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more inform
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: I0708 19:55:15.665304    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more inform
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: I0708 19:55:15.675950    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: I0708 19:55:15.676273    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:31 djg-clusterpi3 kubelet[9233]: I0708 19:55:31.963686    9233 server.go:408] Version: v1.11.0
Jul 08 19:55:31 djg-clusterpi3 kubelet[9233]: I0708 19:55:31.964029    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:31 djg-clusterpi3 kubelet[9233]: I0708 19:55:31.964378    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:31 djg-clusterpi3 kubelet[9233]: I0708 19:55:31.965040    9233 plugins.go:97] No cloud provider specified.






Jul 08 19:55:15 djg-clusterpi3 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more inform
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: I0708 19:55:15.665304    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more inform
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: I0708 19:55:15.675950    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:15 djg-clusterpi3 kubelet[9233]: I0708 19:55:15.676273    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:31 djg-clusterpi3 kubelet[9233]: I0708 19:55:31.963686    9233 server.go:408] Version: v1.11.0
Jul 08 19:55:31 djg-clusterpi3 kubelet[9233]: I0708 19:55:31.964029    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:31 djg-clusterpi3 kubelet[9233]: I0708 19:55:31.964378    9233 feature_gate.go:230] feature gates: &{map[]}
Jul 08 19:55:31 djg-clusterpi3 kubelet[9233]: I0708 19:55:31.965040    9233 plugins.go:97] No cloud provider specified.`

[neolit123]

please, mind that these logs show no errors.

[DJGummikuh]

Yes but that's part of the Issue I believe. I can't find a single conclusive line of [ERROR] anywhere to begin with. That's what's frustrating me so much :-)

Anyways, thanks for formatting that mess of mine :-D

So what would be a good next step?

[neolit123]

So what would be a good next step?

as mentioned previously:

the liveness probe should be made configurable, yet we should probably talk about the best way of doing that in the kubeadm config.

i think these are the values used, so if you are building kubeadm yourself try playing with:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/staticpod/utils.go#L95-L96
but mind that this would increase the timeouts on all control plane components.

[DJGummikuh]

oof I don't build kubeadm by myself, I'm pulling it via apt from apt.kubernetes.io. I don't have anything that remotely resembles a build pipeline for go on any of my machines (never tinkered with it yet). I hoped that there would be a similar file to modify when joining a cluster like there was for creating it, however as it seems these values are hard-coded into the utils.go :-|

[neolit123]

you can try this solution, but it's tricky:
https://github.com/kubernetes/kubeadm/issues/413#issuecomment-402916173

the problem is that this needs a configuration change and i don't think it can be included in a 1.11.X release (but we can try). it also has to be discussed first.

[DJGummikuh]

That's what I already did in the comment where I told you that the master is up (that's what my watch -n 1.0 command was doing). What's happening now is that kubeadm join does not work, and as far as I can see, kubeadm join does not place yaml files for me to patch anywhere :-/

[neolit123]

you could be experiencing another issue. hard to say.

[anguslees]

these are the values used, so if you are building kubeadm yourself try playing with:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/staticpod/utils.go#L95-L96

So I note the InitialDelaySeconds timeout here is _still_ 15s a year later, and I don't understand why it hasn't been increased to something that actually represents reality. Is this bug report serving any purpose at all? I didn't initially propose a PR to just change this number myself because I felt someone already in the kubeadm inner circle could get that done and merged in a few minutes, but I am happy to do so if "lack of a PR" is the only reason we haven't moved forward.

I feel like everyone is trying to come up with reasons to declare the original bug report invalid (perhaps we shouldn't support rPi2, perhaps we should make the initialdelay configurable, perhaps we should pre-pull images or some other unrelated change, perhaps a fast opaque timeout failure is a better UX than a slower success), instead of just increasing the initialDelay timeout to reflect the actual initialDelay that our binaries clearly exhibit and then moving on to something else that actually deserves discussion.

[neolit123]

So I note the InitialDelaySeconds timeout here is still 15s a year later, and I don't understand why it hasn't been increased to something that actually represents reality. Is this bug report serving any purpose at all? I didn't initially propose a PR to just change this number myself because I felt someone already in the kubeadm inner circle could get that done and merged in a few minutes, but I am happy to do so if "lack of a PR" is the only reason we haven't moved forward.

i cannot answer your questions as the whole issue is new to me, but we can try to talk about it this week. please note the team behind kubeadm is a small team with big goals and often there isn't anyone to work on a certain task, like this one.

I feel like everyone is trying to come up with reasons to declare the original bug report invalid (perhaps we shouldn't support rPi2, perhaps we should make the initialdelay configurable, perhaps we should pre-pull images or some other unrelated change, perhaps a fast opaque timeout failure is a better UX than a slower success), instead of just increasing the initialDelay timeout to reflect the actual initialDelay that our binaries clearly exhibit and then moving on to something else that actually deserves discussion.

yes, options have been discussed already in this thread. it's a matter of picking the right option and doing the implementation.

[DJGummikuh]

I actually believe it would make sense to default to "no timeout" with the option of setting a timeout for the whole process (as was suggested somewhere earlier in this issue).

Reason is that most the use cases I can think of actually don't care if a specific step is executed in X Seconds or not since all that is cared about in a distributed system is eventual consistency (spooling up another node just in case is cheaper than fiddling with the settings).

As an interim solution it would however suffice to actually read the timeout settings for kubeadm join from a configuration file just like the kubeadm init stuff does so that our hack with the inflight timeout replacement works. It's a hack, don't think any different - but a terrible hack is still better than no workaround at all.

I'm personally against trying to "guess" sensible timeouts since guesses can always be wrong, would not really serve a purpose in this case (since coping strategy for elapsing timeouts is simply bailing out) and would make reproduction of errors a pain in the ass since two identical systems could start behaving different for a myriad of reasons.

[neolit123]

@anguslees @DJGummikuh we had a talk about that on the recent SIG meeting.

this is a thorny problem, but here are some random notes bellow.

notes:

• some of our people have a lot of experience and they are saying that this smells like a race condition that stalls the apiserver start. it shouldn't take that long. this could be an ARM vs GOLANG thing.
• i've asked the SIG API Machinery but got no response on suggested liveness probe values and if they suspect a different type of issue here.
• we've agreed that exposing a config option or environment variable or command line parameter is a bad idea for kubeadm. the reasoning behind that is that we don't want to expose parameters which can have completely arbitrary meaning to users.
• nobody on the team has ever seen such an issue on slow machines (e.g. slow VMs), which means that this could be RPi related and the whole argument that this is due to "slow" hardware is invalid.
• have a look at this: https://github.com/kubernetes/kubernetes/issues/64357 the user isn't reporting a liveness probe issue at all. any idea why is that?
• Has anyone seen a problem on something like https://www.scaleway.com/, according to the theory in this thread it should happen there too?
• as a bottom line, we've discussed that hardcoding bigger timeout / delay values wouldn't matter too much for kubeadm, so if someone wants to submit a PR for that, go ahead. (@anguslees )

[DJGummikuh]

have a look at this: kubernetes/kubernetes#64357 the user isn't reporting a liveness probe issue at all. any idea why is that?

No it even doesn't seem really reproducible on my hardware now. Since the token for joining nodes elapsed I thought "what the heck" and kubeadm reset'd my cluster master and tried to re-init it (having that watch workaround running) - now even with that workaround I no longer am able to pull up a master on my Raspberry Pi 3+. I even increased the configured timeout from 180 to 300 seconds without any difference. So I like the idea of this being a race condition.
Still I welcome your suggestion to pull request higher timeouts. Won't hurt a lot and gives the pi (which IS a slow hardware, I think we all can agree on that :-) ) a bit more breathing space.

[neolit123]

related issue on ARM64 in the apiserver:
https://github.com/kubernetes/kubernetes/issues/64649

[DJGummikuh]

Did some more research with my (still failing :-( ) Pi Cluster over the weekend.

I reinstalled one of my nodes since I suspected that maybe updating from a Pi 2 to a Pi 3+ without reinstalling the OS might cause some issues. This is not the case, the problem is the same with the brand new OS.
Additionally, I took the effort to compile kubernetes with the increased timeouts and tested with that. Also, this did not really yield any result.
I was able to get the master up (with my watch workaround to patch the configuration files) but joining the cluster with the second node just never succeeds.

It would really be neat to have some additional useful log output to nail this issue down. However I do not understand enough of how the kubernetes components interact to know where to add lines.

Anyone up for the challenge? ^^

[neolit123]

this is really a problem outside kubeadm...

the api machinery folks did not look at my request for comments on the issue, so unless a ticket is logged for this already someone should log one in the kubernetes/kubernetes repo and tag /sig api-machinery.

It would really be neat to have some additional useful log output to nail this issue down. However I do not understand enough of how the kubernetes components interact to know where to add lines.

as a start one can enable --v 4 for the kubelet in the systemd drop-in file, which will tell the GLOG logger to enable high verbosity.
one can also do the same for control plane components from the kubeadm config with this:
https://kubernetes.io/docs/setup/independent/control-plane-flags/

[joejulian]

This solved the problem on my Raspberry Pi 3 cluster: https://github.com/kubernetes/kubernetes/pull/66264

[DJGummikuh]

@joejulian nice I managed to patch that in and now also my cluster is firing up. FINALLY, after weeks of agony! Thank you :-)

[neolit123]

@joejulian thanks for investigating!
a potential fix can be found in https://github.com/kubernetes/kubernetes/pull/66264

closing until further notice.

/close

[ulm0]

is there a way to pass such settings in kubeadm init file? maybe in apiServerExtraArgs? it's a pain to be watching files for patching them, kinda hard to automate.

[joejulian]

There is not. Perhaps that would be a good feature to add.

Later updates have added even more things to check and the extended timeout my PR provided was no longer sufficient. I've given up on managing the timeout. The solution for me was to use ecdsa certificates.

Additionally the controller services including etcd take up more ram, now, than a Raspberry Pi has so rather than double the number of nodes to host the control plane, I've upgraded to Rock64s.

Excuse the pun, but my control plane has been rock solid ever since.

[codesqueak]

I've just been trying to do an install on a Raspberry Pi 3+ and can confirm that the install does indeed fail. Using the 'watch' trick on the kube-apiserver.yaml listed above does seem to work consistently ... but only if I change the initialDelaySeconds to 360. The suggested value of 180 seems marginal on my machines.

Just when things where getting too easy, kubeadm is now complaining that the version of Docker (18.09) is unsupported. A quick revert to 18.06 fixed the issue.

[neolit123]

in kubeadm 1.13 we are adding configuration flag under ClusterConfig->ApiServer that can control the api server timeout.
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1
timeoutForControlPlane

[anguslees]

configuration flag under ClusterConfig->ApiServer that can control the api server timeout.

Searching through the codebase for TimeoutForControlPlane, I think this defaults to 4min, and is only used for the delay used by kubeadm itself to wait for the apiserver to become healthy. In particular, it does _not_ alter the apiserver livenessprobe used by kubelet itself. Is that correct?

I don't think I've seen a counter-argument raised anywhere in the discussion around this issue. Is there a reason we _don't_ just increase livenessProbe initialDelaySeconds and move on to some other problem?

Aside: As far as I can see from a quick read, TimeoutForControlPlane also doesn't take into account other non-failure causes for increased apiserver startup delay, like congestion while pulling multiple images, or additional timeout+retry loop iterations while everything is converging at initial install-time (timeout+retry repeatedly is the k8s design pattern ... and this happens sometimes on a loaded system, which is expected and just fine). I personally feel like 4minutes is both too long for impatient interactive users expecting a fast failure, and too short for an install process on a loaded/slow/automated system that is prepared to wait longer for expected success. How was this arrived at, can we default to 5mins? Can we keep retrying until SIGINT? Why are we imposing an artificial wall-clock deadline internally rather than inheriting it from the calling environment?

Afaics TimeoutForControlPlane is just exposing an arbitrary fatal internal deadline as a parameter where the only intended UX is just to increase the parameter until the deadline is never reached. Alternatively, we could just _not_ impose that arbitrary fatal internal deadline in the first place...

[joejulian]

That's an excellent point and I wholeheartedly agree.

[neolit123]

In particular, it does not alter the apiserver livenessprobe used by kubelet itself. Is that correct?

yes, there are no plans to modify the liveness probes from kubeadm, yet.

this rpi issue was qualified at a sig-cluster-lifecyle meeting as "something that should not happen", "seems almost like a race condition in the kubelet", "why does it only happen on rpi and not on other slower devices". i have to admit i haven't tested slow devices myself.

i.e. there was an agreement that increasing this value:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/staticpod/utils.go#L97
is not a good fix and it seems like a workaround for another bug.

How was this arrived at, can we default to 5mins?

the timeout was 30 minutes before being 4 minutes, because it took pulling images into consideration before 1.11.
if you have an opinion on 4 vs 5 minutes, a well outlined PR with strong points on the subject might make it in 1.14.

[anguslees]

this rpi issue was qualified at a sig-cluster-lifecyle meeting as "something that should not happen", "seems almost like a race condition in the kubelet", "why does it only happen on rpi and not on other slower devices".

Those are all reasons to continue looking for another bug somewhere else as well, but none of those are reasons _not_ to increase initialDelaySeconds. Is there actually a downside to increasing initialDelaySeconds?

To approach it from another direction, if we know we have an issue elsewhere in kubernetes with a workaround that can be used in kubeadm - is it kubeadm's role to "hold the line" on purity and produce a known-broken result? That seems to conflict with the goal to be a tool that we expect people to actually use for actual deployments. So far I've been unable to use kubeadm on my cluster without patching it to increase timeouts (despite reporting it, with patches, over a year ago), which makes me sad.

(Apologies for letting some of my frustration around this issue leak into my tone)

if you have an opinion on 4 vs 5 minutes

Sigh. I was trying to make the case for _no_ timeout in kubeadm, but I have yet to find a way to phrase that proposal convincingly (see this and other failed attempts in this issue, for example) :(

[neolit123]

Those are all reasons to continue looking for another bug somewhere else as well, but none of those are reasons not to increase initialDelaySeconds. Is there actually a downside to increasing initialDelaySeconds?

it is a small change, but it was agreed to not add this increase because it will also apply to systems that don't exercise the problem.

To approach it from another direction, if we know we have an issue elsewhere in kubernetes with a workaround that can be used in kubeadm - is it kubeadm's role to "hold the line" on purity and produce a known-broken result? That seems to conflict with the goal to be a tool that we expect people to actually use for actual deployments.

facing the end user is a goal for kubeadm, that's true.
but again it's only a report for rpis and the actual bug should be escalated to sig-api-machinery (api server) and sig-node (kubelet) and possibly reproduced outside of kubeadm.

So far I've been unable to use kubeadm on my cluster without patching it to increase timeouts (despite reporting it, with patches, over a year ago), which makes me sad.

you don't have to patch kubeadm, you can patch manifest files instead.
kubeadm 1.13 graduates the init phases to top level commands - e.g. kubeadm init phase [phase-name]
phases are there mostly because users want custom control of how the control plane node is created..

if you do kubeadm init --help it will show you in what order the phases are executed.

so you can break down your kubeadm init command into the appropriate phases, you use custom manifests for the control plane components and skip the control-plane phase. there is also --skip-phases now.

you can already do that in 1.11 and 1.12, but it's less straight forward.

[anguslees]

because it will also apply to systems that don't exercise the problem.

So .. what's wrong with that? We fix bugs all the time that only trigger on some systems. Anywhere we have timeouts, we're going to need to tune them for our slowest-ever system, not just some subset of our environments, no?

Another angle on this is that as an ops guy, I'm terrified of cascading failures in overload situations, particularly with the apiserver itself. Afaics, the livenessprobe timeout should only ever trigger when things have clearly failed, not just when it deviates from someone's idea of "normal". Afaics we _should_ have a very relaxed livenessprobe configured, even on our fastest hardware. My little rpi is just demonstrating this overload failure more easily - but it also applies to bigger servers under bigger overload/DoS scenarios. There is no upside to having a small initialDelaySeconds. kubeadm's default livenessprobe is unnecessarily aggressive on all platforms.

I'm sorry I keep repeating the same points, but afaics there are strong practical and theoretical reasons to extend initialDelaySeconds, and I'm just not understanding the opposing argument for keeping it small :(

[neolit123]

adding a kubeadm configuration option for this is unlikely at this point.

i'm trying to explain that this is already doable with 3 commands in 1.13:

sudo kubeadm reset -f
sudo kubeadm init phase control-plane all --config=testkubeadm.yaml
sudo sed -i 's/initialDelaySeconds: 15/initialDelaySeconds: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml
sudo kubeadm init --skip-phases=control-plane --ignore-preflight-errors=all --config=testkubeadm.yaml

[anguslees]

I don't want an option, I'm trying to say the current fixed value (15) should be changed to a different fixed value (360 was suggested above).

.. But I don't want to drag this out any further. It's clear that the decision is to stick with the current value, so I will withdraw defeated. Thanks for your patience :)

[alexellis]

@neolit123 that combination looks great - far easier than what I'd documented - having to wait for the control plane set then quickly running sed in another terminal. https://github.com/alexellis/k8s-on-raspbian/blob/master/GUIDE.md

I'll test the instructions and look to update the guide.

[alexellis]

@neolit123 this is what I got using the config above on an RPi3 B+

[certs] apiserver serving cert is signed for DNS names [rnode-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.110 192.168.0.26 192.168.0.26]
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xaa7204]

goroutine 1 [running]:
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.validateKubeConfig(0xfa93f2, 0xf, 0xfb3d32, 0x17, 0x4032210, 0x68f, 0x7bc)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:236 +0x120
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.createKubeConfigFileIfNotExists(0xfa93f2, 0xf, 0xfb3d32, 0x17, 0x4032210, 0x0, 0xf7978)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:257 +0x90
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.createKubeConfigFiles(0xfa93f2, 0xf, 0x3ec65a0, 0x3f71c60, 0x1, 0x1, 0x0, 0x0)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:120 +0xf4
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.CreateKubeConfigFile(0xfb3d32, 0x17, 0xfa93f2, 0xf, 0x3ec65a0, 0x1f7a701, 0xb9772c)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:93 +0xe8
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases.runKubeConfigFile.func1(0xf66a80, 0x4208280, 0x0, 0x0)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/kubeconfig.go:155 +0x168
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1(0x3cc2d80, 0x0, 0x0)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235 +0x160
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll(0x3ec9270, 0x3f71d68, 0x4208280, 0x0)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:416 +0x5c
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run(0x3ec9270, 0x24, 0x416bdb4)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:208 +0xc8
k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1(0x3e97b80, 0x3e900e0, 0x0, 0x3)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:141 +0xfc
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0x3e97b80, 0x3e3ff80, 0x3, 0x4, 0x3e97b80, 0x3e3ff80)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x20c
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x3e96140, 0x3e97b80, 0x3e96780, 0x3d82100)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x210
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0x3e96140, 0x3c8c0c8, 0x116d958)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x1c
k8s.io/kubernetes/cmd/kubeadm/app.Run(0x3c9c030, 0x0)
    /workspace/anago-v1.13.1-beta.0.57+eec55b9ba98609/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:48 +0x1b0
main.main()
    _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:29 +0x20

kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:36:44Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/arm"}

In kubeadm-config.yaml - 192.168.0.26 is an LB pointing to 192.168.0.110

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
  certSANs:
  - "192.168.0.26"
controlPlaneEndpoint: "192.168.0.26:6443"

[alexellis]

I get the same even without the external config/lb IP.

Alex

[kfox1111]

I've been pushing folks to use kubeadm for a while, even schools wanting to run it on their pi clusters. While I understand not wanting to complicate the code base, I think its probably a good thing for your user base to support running on these little devices. it allows young folks to kick the Kubernetes tires on cheep hardware that otherwise might not. The workaround above, while valid, is much harder for this use case.

What about a compromise? Instead of making it configurable, add a simple heuristic that says, if not x86_64, set the default timeout higher?

[neolit123]

[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xaa7204]

odd, the admin.conf is machine generated and should contain a valid Config type with a current-context filed pointing to a context.

the panic comes from this line:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go#L236

[stephenmoloney]

I'm seeing the exact same error as :point_up: above with the following:

modify_kube_apiserver_config(){
  sed -i 's/failureThreshold: [0-9]/failureThreshold: 15/g' /etc/kubernetes/manifests/kube-apiserver.yaml && \
  sed -i 's/timeoutSeconds: [0-9][0-9]/timeoutSeconds: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml && \
  sed -i 's/initialDelaySeconds: [0-9][0-9]/initialDelaySeconds: 120/g' /etc/kubernetes/manifests/kube-apiserver.yaml
}
kubeadm init phase control-plane all --config=$${KUBEADM_CONFIG_FILE} && \
modify_kube_apiserver_config && \
kubeadm init \
--skip-phases=control-plane \
--ignore-preflight-errors=all \
--config=$${KUBEADM_CONFIG_FILE} \
--v 4

The following script does solve the issue for me using kubeadm versions 1.12, 1.13 (most of the time)

modify_kube_apiserver_config(){
  while [[ ! -e /etc/kubernetes/manifests/kube-apiserver.yaml ]]; do
    sleep 0.5s;
  done && \
  sed -i 's/failureThreshold: [0-9]/failureThreshold: 18/g' /etc/kubernetes/manifests/kube-apiserver.yaml && \
  sed -i 's/timeoutSeconds: [0-9][0-9]/timeoutSeconds: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml && \
  sed -i 's/initialDelaySeconds: [0-9][0-9]/initialDelaySeconds: 240/g' /etc/kubernetes/manifests/kube-apiserver.yaml
}

# ref https://github.com/kubernetes/kubeadm/issues/413 (initialDelaySeconds is too eager)
if [[ ${var.arch} == "arm" ]]; then modify_kube_apiserver_config & fi

kubeadm init \
  --config=$${KUBEADM_CONFIG_FILE} \
  --v ${var.kubeadm_verbosity}

[arnaubennassar]

I was in the same situation, getting the same error with the approach suggested by @neolit123 .
I wasn't able to run the script by @stephenmoloney , I'm not really familiar with bash scripting, probably my fault.

So I ported the script to python (which is installed by default on Raspbian, so no need for extra dependencies), in case anyone is interested:

import os
import time
import threading

filepath = '/etc/kubernetes/manifests/kube-apiserver.yaml'

def replace_defaults():
    print('Thread start looking for the file')
    while not os.path.isfile(filepath):
        time.sleep(1) #wait one second
    print('\033[94m -----------> FILE FOUND: replacing defaults \033[0m')
    os.system("""sed -i 's/failureThreshold: [0-9]/failureThreshold: 18/g' /etc/kubernetes/manifests/kube-apiserver.yaml""")
    os.system("""sed -i 's/timeoutSeconds: [0-9][0-9]/timeoutSeconds: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml""")
    os.system("""sed -i 's/initialDelaySeconds: [0-9][0-9]/initialDelaySeconds: 240/g' /etc/kubernetes/manifests/kube-apiserver.yaml""")

t = threading.Thread(target=replace_defaults)
t.start()
os.system("kubeadm init")

To run it: sudo python however_you_name_the_file.py
Thank you for pointing me to the solution, @stephenmoloney and @neolit123 !

[nelsonyaccuzzi]

Hi! this issue was of much help

I found a fancy way to resolve this using kustomize

mkdir /tmp/kustom

cat > /tmp/kustom/kustomization.yaml <<EOF
patchesJson6902:
- target:
    version: v1
    kind: Pod
    name: kube-apiserver
    namespace: kube-system
  path: patch.yaml
EOF

cat > /tmp/kustom/patch.yaml <<EOF
- op: replace
  path: /spec/containers/0/livenessProbe/initialDelaySeconds
  value: 30
- op: replace
  path: /spec/containers/0/livenessProbe/timeoutSeconds
  value: 30
EOF

sudo kubeadm init --config config.yaml -k /tmp/kustom