Minikube: Stuck at "Stating Cluster Components": failed to "StartContainer" for "etcd" with CrashLoopBackOff

[vinceskahan]

BUG REPORT

Environment:

• Minikube version: v0.27.0
• OS: Mac High Sierra 10.13.4
• VM Driver: virtualbox
• ISO version: v0.26.0
• Install tools:
• Others:
  docker version 18.03.1-ce is installed here too
  vagrant version 2.0.3 is installed
  virtualbox version 5.2.8 r121009 is installed

What happened:
startup hangs infinitely

What you expected to happen:
startup should work within a minute or two max

How to reproduce it (as minimally and precisely as possible):
minikube start

Output of 'minikube logs':
attached

Anything else do we need to know:

This problem was reported for 0.26.x in https://github.com/kubernetes/minikube/issues/2765 but you closed it for some unknown reason after a number of people reported 'me too' on multiple platforms, and some of the other folks mentioned workarounds using flags added to the 'minikube start' command that worked for them.

This is the same issue in a new topic, as requested for some reason. The same issue occurs on the Mac for any minikube version greater than 0.25.2, most recently (re)tested and failing again with 0.27.0

The (deprecated) '--bootstrapper=localkube' workaround mentioned by other folks in #2765 "did" result in my system working ok, FWIW, in case that matters.

Logs for working 0.25.2 and not-working 0.27.0 are attached.

Lastly, I cannot explain why the ISO version in config.json reports as 0.26.0 when 'minikube version' reports 0.27.0 - is it possible it's downloading the wrong version of the ISO for some reason ?

Regardless, my expectation is a simple 'minikube start' should work with no additional flags required, per your minikube docs online.

logs.v0.25.2.txt
logs.v0.27.0.txt

[dlorenc]

From the logs it looks like etcd is crashing at startup time. The iso version mismatch is interesting, but I'm not sure off the top of my head how that would be causing etcd to fail.

[shen0834]

Same issue,from minikube update v0.27.0

[DavidNorena]

+1

[seborama]

Hi. Did you also perform the rm -rf ~/.minikube ~/.kube before starting minikube 0.27 as you did with minikube 0.25? Also check that the VM is correctly deleted in VBox and that no remnant VBox process is running on your computer.

[vinceskahan]

Yes to all.

On May 28, 2018 2:31 PM, "Seb C" [e-mail] wrote:

Hi. Did you also perform the rm -rf ~/.minikube ~/.kube before starting
minikube 0.27 as you did with minikube 0.25? Also check that the VM is
correctly deleted in VBox and that no remnant VBox process is running on
your computer.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/minikube/issues/2841#issuecomment-392606184,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAk2Kn_dj08Nnp7euBQPDGWSgbclXZVEks5t3GyagaJpZM4UNHfU
.

[seborama]

Compared to when I run minikube start (using Hyperkit, I don't have Vbox installed at the moment), the only message that stands out in difference is that you are getting:

net/http: TLS handshake timeout

Whereas I'm getting:

getsockopt: connection refused

Are you using an https proxy or an SSL sniffer such as Charles?
Or, are you on a VPN?

[vinceskahan]

Nope.

On May 28, 2018 2:43 PM, "Seb C" [e-mail] wrote:

Compared to when I run minikube start (using Hyperkit, I don't have Vbox
installed at the moment), the only message that stands out in difference is
that you are getting:

net/http: TLS handshake timeout

Whereas I'm getting:

getsockopt: connection refused

Are you using an https proxy or an SSL sniffer such as Charles?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/minikube/issues/2841#issuecomment-392607515,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAk2Kn-TaKJE0_X4K7jW4rAcN_mQlG_Sks5t3G9ugaJpZM4UNHfU
.

[seborama]

How did you install minikube on your computer?
Do you have port forwarders configured on your VBox (natpf)?
Could you increase the logging?

minikube stop
minikube delete
ps -ef | grep -Ei "vbo[x]|virtualbo[x]" # visually ensure none running
rm -rf ~/.minikube
minikube start --loglevel=0 --logtostderr --v=10

And paste the output here.

PS: never run rm -rf ~/.minikube without first shutting down minikube. If you did, rebooting your Mac is the simplest and most reliable option.

[vinceskahan]

On Mon, May 28, 2018 at 2:46 PM, Seb C [e-mail] wrote:

How did you install minikube on your computer?

For minikube - I followed https://github.com/kubernetes/minikube/releases
using the OS/X curl command there

For kubectl - I followed
https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-via-curl

I did also verify the 'openssl 256sum' checksum of minikube matches what's
on the web page.

--
----- [email protected] ----

[Aschwin89]

I used the command: minikube start --bootstrapper=localkube --vm-driver=no

But this didn't work until I updated VirtualBox to version 5.2.12 (MiniKube v0.27.0).

[w0rd-driven]

The previous issue did kind of jump all over the place which didn't seem to help isolate the underlying cause.

I'm on Sierra and minikube start --vm-driver=hyperkit --bootstrapper=localkube 'worked' in the messages seemed to go through the motions but nothing would ever really start.

What I was just able to work get working successfully for my machine was this instead: minikube start --vm-driver=virtualbox.

I believe that's possibly very important as I have VirtualBox installed on my machine and it's my primary method for using virtual machines. Docker for Mac seems to be set to use virtual box, or specifically I can see the default iso there.

I seem to recall Windows having issues with multiple VM providers in that it can only reliably use one at a time. I don't know if this applies for macOS as well. It's possible this applies to a subset of people but I don't consider this the root cause for everyone affected.

[dlorenc]

For the case where etcd is crashlooping, could you try to provide logs from the etcd container? You should be able to see it in "docker ps -a" or in kubectl get pods --all-namespaces"

[lzlicd]

A bug introduced in 0.26+, Downgrade to 0.25 works. https://github.com/kubernetes/minikube/issues/2765#issuecomment-385690911

[jhagege]

+1

[mpern]

Same here, with minikube version: v0.28.2

[tstromberg]

This is quite possibly a dupe of #2967 (etcd dns issue) - from your logs, I see:

May 24 22:24:07 minikube kubelet[3614]: E0524 22:24:07.295399 3614 pod_workers.go:186] Error syncing pod 53b082a7696cac7320128fcdbfdc22bc ("etcd-minikube_kube-system(53b082a7696cac7320128fcdbfdc22bc)"), skipping: failed to "StartContainer" for "etcd" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=etcd pod=etcd-minikube_kube-system(53b082a7696cac7320128fcdbfdc22bc)"

It's hard to tell without seeing etcd logs though.

[351zyf]

A bug introduced in 0.26+, Downgrade to 0.25 works. #2765 (comment)

use command

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.25.2/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

to install v0.25 fix this.
OSX 10.13

[tstromberg]

Marking this as obsolete, as the versions of minikube, kubernetes, and etcd have all improved tremendously over the last 14 months since this bug was opened. If you are still seeing this, feel free to re-open it. Thanks!