Minikube: Minikube hangs on --extra-config=apiserver.Admission.PluginNames

[mdkess]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Environment: OS X 10.12.6 (Sierra)

Minikube version (use minikube version): v0.24.1

• OS (e.g. from /etc/os-release): Darwin Martin-Kess 16.7.0 Darwin Kernel Version 16.7.0: Mon Nov 13 21:56:25 PST 2017; root:xnu-3789.72.11~1/RELEASE_X86_64 x86_64
• VM Driver: virtualbox
• ISO version: "Boot2DockerURL": "file:///Users/martinkess/.minikube/cache/iso/minikube-v0.23.6.iso",
• Install tools:
• Others:

What happened:

When I try to create a minikube instance with --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota" (for automatic Istio.io side-car injection), the system hangs at the "starting cluster components" step. When I try to start without the admission controllers, minikube starts just fine.

Here's the output of my start command:

minikube start \
  --v=999 --alsologtostderr \
  --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota" \
  --bootstrapper kubeadm \
  --kubernetes-version v1.9.0 \
  --vm-driver=virtualbox

Starting cluster components...
I1222 09:37:22.905972    5433 ssh_runner.go:57] Run: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --skip-preflight-checks
I1222 09:37:23.992206    5433 ssh_runner.go:57] Run: sudo rm -rf /tmp/kube-controller-manager-amd64_v1.9.0
I1222 09:37:24.006906    5433 cache_images.go:219] Successfully loaded image /Users/martinkess/.minikube/cache/images/gcr.io/google_containers/kube-controller-manager-amd64_v1.9.0 from cache
I1222 09:37:26.026285    5433 ssh_runner.go:57] Run: sudo rm -rf /tmp/kubernetes-dashboard-amd64_v1.6.3
I1222 09:37:26.027845    5433 ssh_runner.go:57] Run: sudo rm -rf /tmp/kube-apiserver-amd64_v1.9.0
I1222 09:37:26.050692    5433 cache_images.go:219] Successfully loaded image /Users/martinkess/.minikube/cache/images/gcr.io/google_containers/kubernetes-dashboard-amd64_v1.6.3 from cache
I1222 09:37:26.051629    5433 cache_images.go:219] Successfully loaded image /Users/martinkess/.minikube/cache/images/gcr.io/google_containers/kube-apiserver-amd64_v1.9.0 from cache

And then nothing for 20 minutes.

I connected with gdb:

(gdb) bt
#0  runtime.kevent () at /usr/local/go/src/runtime/sys_darwin_amd64.s:526
#1  0x0000000001028953 in runtime.netpoll (block=true, ~r1=0x1) at /usr/local/go/src/runtime/netpoll_kqueue.go:79
#2  0x000000000103214b in runtime.findrunnable (gp=0xc42002cb90, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2107
#3  0x0000000001032adc in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2245
#4  0x0000000001032df6 in runtime.park_m (gp=0xc420aa0c00) at /usr/local/go/src/runtime/proc.go:2308
#5  0x0000000001058e7b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:286
#6  0x0000000002924800 in crypto/tls.emptyConfig ()
#7  0x00007fff5fbff730 in ?? ()
#8  0x0000000002924840 in crypto/tls.emptyConfig ()
#9  0x00007fff5fbff720 in ?? ()
#10 0x000000000102fd94 in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1142
#11 0x0000000001058d41 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:186
#12 0x00007fff0000000a in ?? ()
#13 0x00007fff5fbff760 in ?? ()
#14 0x000000000000000a in ?? ()
#15 0x00007fff5fbff760 in ?? ()
#16 0x0000000000000000 in ?? ()
(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x1803 of process 5433 runtime.kevent () at /usr/local/go/src/runtime/sys_darwin_amd64.s:526
  2    Thread 0x1903 of process 5433 runtime.mach_semaphore_timedwait () at /usr/local/go/src/runtime/sys_darwin_amd64.s:455
  3    Thread 0x1a03 of process 5433 runtime.mach_semaphore_wait () at /usr/local/go/src/runtime/sys_darwin_amd64.s:445
  4    Thread 0x1b03 of process 5433 runtime.mach_semaphore_wait () at /usr/local/go/src/runtime/sys_darwin_amd64.s:445
  5    Thread 0x1c03 of process 5433 runtime.mach_semaphore_wait () at /usr/local/go/src/runtime/sys_darwin_amd64.s:445
  6    Thread 0x1d03 of process 5433 runtime.mach_semaphore_wait () at /usr/local/go/src/runtime/sys_darwin_amd64.s:445
  7    Thread 0x1e03 of process 5433 runtime.mach_semaphore_timedwait () at /usr/local/go/src/runtime/sys_darwin_amd64.s:455
  8    Thread 0x1f03 of process 5433 runtime.mach_semaphore_wait () at /usr/local/go/src/runtime/sys_darwin_amd64.s:445
  9    Thread 0x2003 of process 5433 runtime.mach_semaphore_wait () at /usr/local/go/src/runtime/sys_darwin_amd64.s:445

Let me know if there's any other useful information that I could provide.

What you expected to happen:

minikube to start.

How to reproduce it (as minimally and precisely as possible):

minikube start \
  --v=999 --alsologtostderr \
  --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota" \
  --bootstrapper kubeadm \
  --kubernetes-version v1.9.0 \
  --vm-driver=virtualbox

[sayanh]

I can also reproduce the same issue with

minikube start \
  --kubernetes-version v1.8.6 \
  --bootstrapper kubeadm \
  --v 10 \
  --logtostderr \
  --vm-driver=virtualbox \
  --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota,PodPreset"

[sayanh]

Ok, I found this solution: the correct config is apiserver.admission-control which does not hang minikube.
E.g

minikube start --cpus 4 --memory 4096 \ --kubernetes-version v1.8.6 \ --bootstrapper kubeadm \ --v 7 \ --logtostderr \ --vm-driver=virtualbox \ --extra-config=apiserver.admission-control="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota,PodPreset"

[r2d4]

https://github.com/kubernetes/minikube/blob/master/docs/configuring_kubernetes.md#kubeadm-bootstrapper