minikube start --extra-config=apiserver.Authorization.Mode=RBAC hangs

[metal3d]

Environment:

Minikube version (use minikube version):

• OS (e.g. from /etc/os-release): Fedora 28, x86_64
• VM Driver: kvm2
• "ISO": "/home/patachou/.minikube/machines/minikube/boot2docker.iso",
  "Boot2DockerURL": "file:///home/patachou/.minikube/cache/iso/minikube-v0.28.1.iso"
• Others:

What happened:
I'm using ./minikube start --extra-config=apiserver.Authorization.Mode=RBAC because I need to test nuclio with traekif, and we need to change RBAC

What you expected to happen:
A started minikube :)

How to reproduce it (as minimally and precisely as possible):
As far as I know, I had minikube working just before to add the RBAC option. After having found that I need that option, I only:

minikube stop
minikube delete
rm -rf ~/.minikube
# and removed all minikube entry found in kubectl config to be sure

./minikube start  --extra-config=apiserver.Authorization.Mode=RBAC --vm-driver=kvm
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
WARNING: The kvm driver is now deprecated and support for it will be removed in a future release.
                Please consider switching to the kvm2 driver, which is intended to replace the kvm driver.
                See https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver for more information.
                To disable this message, run [minikube config set WantShowDriverDeprecationNotification false]
Downloading Minikube ISO
 160.27 MB / 160.27 MB [============================================] 100.00% 0s
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.10.0
Downloading kubelet v1.10.0
Finished Downloading kubelet v1.10.0
Finished Downloading kubeadm v1.10.0
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...

Output of minikube logs (if applicable):

Sep 12 08:12:04 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Sep 12 08:12:04 minikube kubelet[2325]: Flag --allow-privileged has been deprecated, will be removed in a future version
Sep 12 08:12:04 minikube kubelet[2325]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 08:12:04 minikube kubelet[2325]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 08:12:04 minikube kubelet[2325]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.12, and the cadvisor port will be removed entirely in 1.13
Sep 12 08:12:04 minikube kubelet[2325]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 08:12:04 minikube kubelet[2325]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 08:12:04 minikube kubelet[2325]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 08:12:04 minikube kubelet[2325]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 08:12:04 minikube kubelet[2325]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 08:12:04 minikube kubelet[2325]: I0912 08:12:04.174805    2325 feature_gate.go:226] feature gates: &{{} map[]}
Sep 12 08:12:04 minikube kubelet[2325]: F0912 08:12:04.745742    2325 server.go:218] unable to load client CA file /var/lib/localkube/certs/ca.crt: open /var/lib/localkube/certs/ca.crt: no such file or directory
Sep 12 08:12:04 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 12 08:12:04 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 12 08:12:14 minikube systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 12 08:12:14 minikube systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Sep 12 08:12:14 minikube systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Sep 12 08:12:14 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Sep 12 08:12:14 minikube kubelet[2388]: Flag --allow-privileged has been deprecated, will be removed in a future version


[...]


Sep 12 08:19:23 minikube kubelet[2388]: E0912 08:19:23.487945    2388 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Sep 12 08:19:23 minikube kubelet[2388]: E0912 08:19:23.489091    2388 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://localhost:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Sep 12 08:19:23 minikube kubelet[2388]: E0912 08:19:23.490166    2388 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://localhost:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: getsockopt: connection refused

Anything else do we need to know:
I tried bootstrapper "localkube" and it seems to work... but I need to install dashboard manually

I wonder if the problem is the RBAC option... So I reproduced without that option, and it won't start

[metal3d]

Seems to be the same as #2765 #2456 #3117 and so on... It seems that there is a real problem, any help will be appreciated

[tstromberg]

Sorry about this. We need to do a better job of making these issues easier to debug.

Question: your output mentions --vm_driver=kvm, but your environment details say kvm2. Was that a typo?

[tstromberg]

I tried "minikube --extra-config=apiserver.Authorization.Mode=RBAC --vm-driver=kvm2 start"
myself, but it hung for 30 minutes instead of forever and output:

Starting cluster components...
E0918 14:14:23.195503 62677 start.go:300] Error starting cluster: kubeadm init error
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI &&
sudo /usr/bin/kubeadm alpha phase addon kube-dns
running command: : running command:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI &&
sudo /usr/bin/kubeadm alpha phase addon kube-dns

[tstromberg]

Likely dupe of #2852