Minikube: vmware+vbox: kubelet: dial tcp 192.168.99.100:8443: getsockopt: connection refused

[hana-shin]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
This is a BUG REPORT.

Please provide the following details:

Environment:
Windows10 + VMware Workstation 14 Player

Guest machine is working on VMware Workstation 14 Player.
I have installed minikube on guest machine.

Minikube version (use minikube version):

• OS (e.g. from /etc/os-release):
  [root@master ~]# cat /etc/redhat-release
  CentOS Linux release 7.4.1708 (Core)
  [root@master ~]# uname -r
  3.10.0-693.el7.x86_64

• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
  [root@master ~]# cat ~/.minikube/machines/minikube/config.json | grep DriverName
  "DriverName": "virtualbox",

• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):
  [root@master ~]# cat ~/.minikube/machines/minikube/config.json | grep -i ISO
  "Boot2DockerURL": "file:///root/.minikube/cache/iso/minikube-v0.28.0.iso",

• Install tools:

• Others:
  The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):

minikube version
echo "";
echo "OS:";
cat /etc/os-release
echo "";
echo "VM driver": 
grep DriverName ~/.minikube/machines/minikube/config.json
echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:
I have started minikube as follows.
However, kubelet service failed repeatedly every 10 seconds.

[root@master ~]# minikube --vm-driver=virtualbox start
[root@master ~]# cat /var/log/messages |grep "kubelet.service: main process exited"
Jun 23 17:33:34 master systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Jun 23 17:33:44 master systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Jun 23 17:33:54 master systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Jun 23 17:34:05 master systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Jun 23 17:34:15 master systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Jun 23 17:34:25 master systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Jun 23 17:34:35 master systemd: kubelet.service: main process exited, code=exited, status=255/n/a
-snip-

[root@master ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
mq10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since 土 2018-06-23 17:35:27 JST; 6s ago
Docs: http://kubernetes.io/docs/
Process: 13006 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
Main PID: 13006 (code=exited, status=255)

Jun 23 17:35:27 master systemd[1]: Unit kubelet.service entered failed state.
Jun 23 17:35:27 master systemd[1]: kubelet.service failed.

What you expected to happen:
kubelet service should not to be failed.

How to reproduce it (as minimally and precisely as possible):

1. I have enabled Intel VT-x/EPT of guest machine.
2. I have statred guest machine.
3. I have added repository of kubernetes on the guest machine. The content is as follows.
   [root@master ~]# cat /etc/yum.repos.d/kubernetes.repo
   [kubernetes]
   name=Kubernetes
   baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
   enabled=1
   gpgcheck=1
   repo_gpgcheck=1
   gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
4. I have installed kubectl package.
   [root@master ~]# yum install -y kubectl
   [root@master ~]# kubectl version
   Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
   Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
5. I have installed VirtualBox package. The VirtualBox package has been installed successfully.
   [root@master ~]# wget https://download.virtualbox.org/virtualbox/5.2.12/VirtualBox-5.2-5.2.12_122591_el7-1.x86_64.rpm
   [root@master ~]# ls VirtualBox-5.2-5.2.12_122591_el7-1.x86_64.rpm
   VirtualBox-5.2-5.2.12_122591_el7-1.x86_64.rpm
   [root@master ~]# rpm -ivh VirtualBox-5.2-5.2.12_122591_el7-1.x86_64.rpm
   -snip-
   Creating group 'vboxusers'. VM users must be member of that group!

1. I have installed minikube command.
   [root@master ~]# curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

2. I have executed minikube command as follows.
   [root@master ~]# minikube --vm-driver=virtualbox start
   Starting local Kubernetes v1.10.0 cluster...
   Starting VM...
   Downloading Minikube ISO
   153.08 MB / 153.08 MB [============================================] 100.00% 0s
   Getting VM IP address...
   Moving files into cluster...
   Downloading kubeadm v1.10.0
   Downloading kubelet v1.10.0
   Finished Downloading kubeadm v1.10.0
   Finished Downloading kubelet v1.10.0
   Setting up certs...
   Connecting to cluster...
   Setting up kubeconfig...
   Starting cluster components...
   Kubectl is now configured to use the cluster.
   Loading cached images from config file.

3. I have checked the /var/log/messages and kubectl status. I have noticed that kubelet service failed repeatedly every 10 seconds.
   [root@master ~]# cat /var/log/messages
   [root@master ~]# systemctl status kubelet
   ● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Drop-In: /etc/systemd/system/kubelet.service.d
   mq10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since 土 2018-06-23 18:11:19 JST; 8s ago
   Docs: http://kubernetes.io/docs/
   Process: 15714 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
   Main PID: 15714 (code=exited, status=255)

   6月 23 18:11:19 master systemd[1]: Unit kubelet.service entered failed state.
   6月 23 18:11:19 master systemd[1]: kubelet.service failed.

Output of minikube logs (if applicable):
-- Logs begin at Sat 2018-06-23 06:26:31 UTC, end at Sat 2018-06-23 08:38:01 UTC. --
Jun 23 06:26:58 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jun 23 06:26:58 minikube kubelet[2631]: Flag --pod-manifest-path has been deprecated, This parameter should be set via th
e config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-c
onfig-file/ for more information.
Jun 23 06:26:58 minikube kubelet[2631]: Flag --allow-privileged has been deprecated, will be removed in a future version
Jun 23 06:26:58 minikube kubelet[2631]: Flag --cluster-domain has been deprecated, This parameter should be set via the c
onfig file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-conf
ig-file/ for more information.
Jun 23 06:26:58 minikube kubelet[2631]: Flag --authorization-mode has been deprecated, This parameter should be set via t
he config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-
config-file/ for more information.
Jun 23 06:26:58 minikube kubelet[2631]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled)
in 1.12, and the cadvisor port will be removed entirely in 1.13
Jun 23 06:26:58 minikube kubelet[2631]: Flag --fail-swap-on has been deprecated, This parameter should be set via the con
fig file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config
-file/ for more information.
Jun 23 06:26:58 minikube kubelet[2631]: Flag --client-ca-file has been deprecated, This parameter should be set via the c
onfig file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-conf
ig-file/ for more information.
Jun 23 06:26:58 minikube kubelet[2631]: Flag --cgroup-driver has been deprecated, This parameter should be set via the co
nfig file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-confi
g-file/ for more information.
Jun 23 06:26:58 minikube kubelet[2631]: Flag --cluster-dns has been deprecated, This parameter should be set via the conf
ig file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-
file/ for more information.
Jun 23 06:26:58 minikube kubelet[2631]: I0623 06:26:58.441399 2631 feature_gate.go:226] feature gates: &{{} map[]}
Jun 23 06:26:58 minikube kubelet[2631]: W0623 06:26:58.563427 2631 hostport_manager.go:68] The binary conntrack is not
installed, this can cause failures in network connection cleanup.
Jun 23 06:26:58 minikube kubelet[2631]: I0623 06:26:58.563708 2631 server.go:376] Version: v1.10.0
Jun 23 06:26:58 minikube kubelet[2631]: I0623 06:26:58.563979 2631 feature_gate.go:226] feature gates: &{{} map[]}
Jun 23 06:26:58 minikube kubelet[2631]: I0623 06:26:58.564479 2631 plugins.go:89] No cloud provider specified.
Jun 23 06:26:58 minikube kubelet[2631]: F0623 06:26:58.564677 2631 server.go:233] failed to run Kubelet: unable to loa
d bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
Jun 23 06:26:58 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jun 23 06:26:58 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jun 23 06:27:08 minikube systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jun 23 06:27:08 minikube systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Jun 23 06:27:08 minikube systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jun 23 06:27:08 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jun 23 06:27:09 minikube kubelet[2684]: Flag --pod-manifest-path has been deprecated, This parameter should be set via th
e config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-c
onfig-file/ for more information.
Jun 23 06:27:09 minikube kubelet[2684]: Flag --allow-privileged has been deprecated, will be removed in a future version
Jun 23 06:27:09 minikube kubelet[2684]: Flag --cluster-domain has been deprecated, This parameter should be set via the c
onfig file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:09 minikube kubelet[2684]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:09 minikube kubelet[2684]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.12, and the cadvisor port will be removed entirely in 1.13
Jun 23 06:27:09 minikube kubelet[2684]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:09 minikube kubelet[2684]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:09 minikube kubelet[2684]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:09 minikube kubelet[2684]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:09 minikube kubelet[2684]: I0623 06:27:09.026571 2684 feature_gate.go:226] feature gates: &{{} map[]}
Jun 23 06:27:09 minikube kubelet[2684]: W0623 06:27:09.068436 2684 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
Jun 23 06:27:09 minikube kubelet[2684]: I0623 06:27:09.068628 2684 server.go:376] Version: v1.10.0
Jun 23 06:27:09 minikube kubelet[2684]: I0623 06:27:09.068673 2684 feature_gate.go:226] feature gates: &{{} map[]}
Jun 23 06:27:09 minikube kubelet[2684]: I0623 06:27:09.068946 2684 plugins.go:89] No cloud provider specified.
Jun 23 06:27:09 minikube kubelet[2684]: F0623 06:27:09.068990 2684 server.go:233] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
Jun 23 06:27:09 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jun 23 06:27:09 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jun 23 06:27:19 minikube systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jun 23 06:27:19 minikube systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 2.
Jun 23 06:27:19 minikube systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jun 23 06:27:19 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jun 23 06:27:19 minikube kubelet[2717]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:19 minikube kubelet[2717]: Flag --allow-privileged has been deprecated, will be removed in a future version
Jun 23 06:27:19 minikube kubelet[2717]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:19 minikube kubelet[2717]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:19 minikube kubelet[2717]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.12, and the cadvisor port will be removed entirely in 1.13
Jun 23 06:27:19 minikube kubelet[2717]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:19 minikube kubelet[2717]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:19 minikube kubelet[2717]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:19 minikube kubelet[2717]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.487994 2717 feature_gate.go:226] feature gates: &{{} map[]}
Jun 23 06:27:19 minikube kubelet[2717]: W0623 06:27:19.529746 2717 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.530188 2717 server.go:376] Version: v1.10.0
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.530762 2717 feature_gate.go:226] feature gates: &{{} map[]}
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.531627 2717 plugins.go:89] No cloud provider specified.
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.692982 2717 server.go:613] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.694036 2717 container_manager_linux.go:242] container manager verified user specified cgroup-root exists: /
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.694186 2717 container_manager_linux.go:247] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:} {Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.1} GracePeriod:0s MinReclaim:} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.15} GracePeriod:0s MinReclaim:}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true}
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.698311 2717 container_manager_linux.go:266] Creating device plugin manager: true
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.698457 2717 state_mem.go:36] [cpumanager] initializing new in-memory state store
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.700007 2717 state_mem.go:87] [cpumanager] updated default cpuset: ""
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.700179 2717 state_mem.go:95] [cpumanager] updated cpuset assignments: "map[]"
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.700365 2717 kubelet.go:272] Adding pod path: /etc/kubernetes/manifests
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.701765 2717 kubelet.go:297] Watching apiserver
Jun 23 06:27:19 minikube kubelet[2717]: E0623 06:27:19.710061 2717 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.99.100:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube
&limit=500&resourceVersion=0: dial tcp 192.168.99.100:8443: getsockopt: connection refused
Jun 23 06:27:19 minikube kubelet[2717]: E0623 06:27:19.713745 2717 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.99.100:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.99.100:8443: getsockopt: connection refused
Jun 23 06:27:19 minikube kubelet[2717]: E0623 06:27:19.719160 2717 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.99.100:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.99.100:8443: getsockopt: connection refused
Jun 23 06:27:19 minikube kubelet[2717]: W0623 06:27:19.720662 2717 kubelet_network.go:139] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.720872 2717 kubelet.go:556] Hairpin mode set to "hairpin-veth"
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.722704 2717 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.722944 2717 client.go:104] Start docker client with request timeout=2m0s
Jun 23 06:27:19 minikube kubelet[2717]: W0623 06:27:19.742335 2717 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.745527 2717 docker_service.go:244] Docker cri networking managed by kubernetes.io/no-op
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.776571 2717 docker_service.go:249] Docker Info: &{ID:ZGJS:SCGS:HEQG:EXO4:ESI3:TCWO:Z4R6:MSFP:AXB7:N2RG:VQKT:5KZT Containers:30 ContainersRunning:1 ContainersPaused:0 ContainersStopped:29 Images:13 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:22 OomKillDisable:true NGoroutines:35 SystemTime:2018-06-23T06:27:19.750937909Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.16.14 OperatingSystem:Buildroot 2018.05 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc4206cbea0 NCPU:2 MemTotal:2088189952 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:minikube Labels:[provider=virtualbox] ExperimentalBuild:false ServerVersion:17.12.1-ce ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:docker-runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:9b55aab90508bd389d7654c4baf173a981477d55 Expected:9b55aab90508bd389d7654c4baf173a981477d55} RuncCommit:{ID:9f9c96235cc97674e935002fc3d78361b696a69e Expected:9f9c96235cc97674e935002fc3d78361b696a69e} InitCommit:{ID:N/A Expected:} SecurityOptions:[name=seccomp,profile=default]}
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.779740 2717 docker_service.go:262] Setting cgroupDriver to cgroupfs
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.912497 2717 remote_runtime.go:43] Connecting to runtime service unix:///var/run/dockershim.sock
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.961260 2717 kuberuntime_manager.go:186] Container runtime docker initialized, version: 17.12.1-ce, apiVersion: 1.35.0
Jun 23 06:27:19 minikube kubelet[2717]: W0623 06:27:19.961730 2717 probe.go:215] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.966014 2717 csi_plugin.go:61] kubernetes.io/csi: plugin initializing...
Jun 23 06:27:19 minikube kubelet[2717]: E0623 06:27:19.996706 2717 kubelet.go:1277] Image garbage collection failed on
un 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.997596 2717 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.997839 2717 status_manager.go:140] Starting to sync pod status with apiserver
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.997857 2717 kubelet.go:1777] Starting kubelet main sync loop.
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.997876 2717 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
Jun 23 06:27:19 minikube kubelet[2717]: I0623 06:27:19.998037 2717 server.go:129] Starting to listen on 0.0.0.0:10250
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:19.999557 2717 server.go:299] Adding debug handlers to kubelet server.
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:20.004038 2717 volume_manager.go:247] Starting Kubelet Volume Manager
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:20.004532 2717 desired_state_of_world_populator.go:129] Desired state populator starts to run
Jun 23 06:27:20 minikube kubelet[2717]: E0623 06:27:20.010438 2717 event.go:209] Unable to write event: 'Post https://192.168.99.100:8443/api/v1/namespaces/default/events: dial tcp 192.168.99.100:8443: getsockopt: connection refused' (may retry after sleeping)
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:20.014486 2717 server.go:944] Started kubelet
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:20.128021 2717 kubelet.go:1794] skipping pod synchronization - [container runtime is down]
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:20.128264 2717 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:20.150689 2717 kubelet_node_status.go:82] Attempting to register node minikube
Jun 23 06:27:20 minikube kubelet[2717]: E0623 06:27:20.151275 2717 kubelet_node_status.go:106] Unable to register node "minikube" with API server: Post https://192.168.99.100:8443/api/v1/nodes: dial tcp 192.168.99.100:8443: getsockopt: connection refused
Jun 23 06:27:20 minikube kubelet[2717]: E0623 06:27:20.292774 2717 remote_runtime.go:278] ContainerStatus "7a6c8b4960c2baf45b4a494ffed66e723da44ca0bd14183d8a8710d35018f084" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 7a6c8b4960c2baf45b4a494ffed66e723da44ca0bd14183d8a8710d35018f084
Jun 23 06:27:20 minikube kubelet[2717]: E0623 06:27:20.293956 2717 kuberuntime_container.go:385] ContainerStatus for 7a6c8b4960c2baf45b4a494ffed66e723da44ca0bd14183d8a8710d35018f084 error: rpc error: code = Unknown desc = Error: No such container: 7a6c8b4960c2baf45b4a494ffed66e723da44ca0bd14183d8a8710d35018f084
Jun 23 06:27:20 minikube kubelet[2717]: E0623 06:27:20.294055 2717 kuberuntime_manager.go:873] getPodContainerStatuses for pod "storage-provisioner_kube-system(8773138f-7614-11e8-9a97-080027907ff9)" failed: rpc error: code = Unknown desc = Error: No such container: 7a6c8b4960c2baf45b4a494ffed66e723da44ca0bd14183d8a8710d35018f084
Jun 23 06:27:20 minikube kubelet[2717]: E0623 06:27:20.294128 2717 generic.go:241] PLEG: Ignoring events for pod storage-provisioner/kube-system: rpc error: code = Unknown desc = Error: No such container: 7a6c8b4960c2baf45b4a494ffed66e723da44ca0bd14183d8a8710d35018f084
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:20.329072 2717 kubelet.go:1794] skipping pod synchronization - [container runtime is down]
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:20.352181 2717 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Jun 23 06:27:20 minikube kubelet[2717]: I0623 06:27:20.362265 2717 kubelet_node_status.go:82] Attempting to register n
-snip-

Anything else do we need to know:
[root@master ~]# lscpu |grep vmx
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec arat

[root@master ~]# getenforce
Permissive

[root@master ~]# systemctl is-active firewalld.service
inactive

[root@master ~]# minikube version
minikube version: v0.28.0

[root@master ~]# minikube get-k8s-versions
The following Kubernetes versions are available when using the localkube bootstrapper:
- v1.10.0
- v1.9.4
-snip-

Although kubelet service failed repeatedly every 10 seconds, I can get cluster information.
[root@master ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
KubeDNS is running at https://192.168.99.100:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Although kubelet service failed repeatedly every 10 seconds, I can get node information.
[root@master ~]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready master 3m v1.10.0 Buildroot 2018.05 4.16.14 docker://17.12.1-ce

[shashidharatd]

I am also facing similar issue, which causes lot of restarts to kubelet but eventually it succeeds, not sure how. The main problem kubelet is failing is with the below reason.

Jun 23 06:27:09 minikube kubelet[2684]: F0623 06:27:09.068990 2684 server.go:233] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

[cdancy]

Same here as well.

[harryhazza77]

me too - eventually it did succeed but the log is filled with errors. The dashboard does not load.

[cdancy]

FWIW: I moved to using the oc tool for OpenShift kubernetes as it does exactly what I want the "minikube with none driver" to do without any hassle. Highly recommend it.

[jomach]

same here using macosx with fresh install
minikube version: v0.31.0
VirtualBox Version 5.2.22 r126460 (Qt5.6.3)

I solved it by deleting the images and my .minikube folder

[bsingarayan]

I too get the same error on macosx
minikube version: v0.28.0
VirtualBox Version: 5.2.6r120293

Any idea how to fix this? I did remove .minikube directory before restarting minikube.

[jomach]

Update minikube. Delete the images and delete the .minikube. Should work then

[mpictor]

I updated minikube from 0.31.0 to 0.32.0 and cleaned up, then tried again. Still does _not_ work for me.
cleanup:

minikube stop
minikube delete
rm -rf ~/.minikube 

~/.minikube contained the images, according to grep -i ISO ~/.minikube/machines/minikube/config.json

minikube start --vm-driver kvm2 still fails
$ minikube logs

...
Jan 06 22:45:17 minikube kubelet[2638]: I0106 22:45:17.471944    2638 server.go:408] Version: v1.12.4
Jan 06 22:45:17 minikube kubelet[2638]: I0106 22:45:17.472549    2638 plugins.go:99] No cloud provider specified.
Jan 06 22:45:17 minikube kubelet[2638]: F0106 22:45:17.472764    2638 server.go:262] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
...

[Pan-Bubr]

I'm having the same problems. I have fresh setup created with all newest pieces of software straight from the brew and/or software webpages.
MacOS: 10.14.2
kubernetes-cli (kubectl): 1.13.1
minikube: 0.32.0
VirtualBox: 6.0

Jan 08 17:28:45 minikube kubelet[2907]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:45 minikube kubelet[2907]: Flag --allow-privileged has been deprecated, will be removed in a future version
Jan 08 17:28:45 minikube kubelet[2907]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:45 minikube kubelet[2907]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:45 minikube kubelet[2907]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:45 minikube kubelet[2907]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:45 minikube kubelet[2907]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:45 minikube kubelet[2907]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:45 minikube kubelet[2907]: I0108 17:28:45.811951    2907 server.go:408] Version: v1.12.4
Jan 08 17:28:45 minikube kubelet[2907]: I0108 17:28:45.812249    2907 plugins.go:99] No cloud provider specified.
Jan 08 17:28:45 minikube kubelet[2907]: F0108 17:28:45.812307    2907 server.go:262] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
Jan 08 17:28:45 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 08 17:28:45 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 08 17:28:55 minikube systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jan 08 17:28:55 minikube systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 3.
Jan 08 17:28:55 minikube systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 08 17:28:55 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 08 17:28:56 minikube kubelet[2952]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:56 minikube kubelet[2952]: Flag --allow-privileged has been deprecated, will be removed in a future version
Jan 08 17:28:56 minikube kubelet[2952]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:56 minikube kubelet[2952]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:56 minikube kubelet[2952]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:56 minikube kubelet[2952]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:56 minikube kubelet[2952]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:56 minikube kubelet[2952]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 08 17:28:56 minikube kubelet[2952]: I0108 17:28:56.076782    2952 server.go:408] Version: v1.12.4
Jan 08 17:28:56 minikube kubelet[2952]: I0108 17:28:56.076988    2952 plugins.go:99] No cloud provider specified.
Jan 08 17:28:56 minikube kubelet[2952]: F0108 17:28:56.077023    2952 server.go:262] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
Jan 08 17:28:56 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 08 17:28:56 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.

[jomach]

@mpictor I was using virtualbox and mac

[Tianny]

So what's the solution?

[jomach]

@Tianny it worked for me doing this:
minikube stop
minikube delete
rm -rf ~/.minikube
But I was using mac and virtualbox

[Tianny]

@jomach Thanks. But I am using mac and hyperkit like this:
minikube stop
minikube delete
rm -rf ~/.minikube/
rm -rf ~/.kube/
minikube start --vm-driver=hyperkit

But still has the error.

[bigknife]

mee too. minikube version: v0.32.0.

removing .kube and .minikube not work.

[Gerromie]

From what I can tell, this appears to be an incompatibility between the components that construct the environment. The work around for me was to switch from Docker Desktop Mac Stable to Edge.

In looking at the version of Kubernetes in Docker Desktop Mac Stable, it was v1.10.0. Edge comes with Kubernetes v1.13.0.

While attempting to start minikube earlier, I had the latest versions of these tools:

Minikube v0.32.0
Kubernetes v1.12.4 (selected by minikube)
kubectl v1.13.2
Virtualbox 6.0.2

I still have the above, but after changing to Docker Desktop Mac (Edge), minikube start ran successfully.

[Tianny]

Docker Desktop Mac

From what I can tell, this appears to be an incompatibility between the components that construct the environment. The work around for me was to switch from Docker Desktop Mac Stable to Edge.

In looking at the version of Kubernetes in Docker Desktop Mac Stable, it was v1.10.0. Edge comes with Kubernetes v1.13.0.

While attempting to start minikube earlier, I had the latest versions of these tools:

Minikube v0.32.0
Kubernetes v1.12.4 (selected by minikube)
kubectl v1.13.2
Virtualbox 6.0.2

I still have the above, but after changing to Docker Desktop Mac (Edge), minikube start ran successfully.

That's cool.

[ShalbafZadeh]

this happens to me too on linux
```
[reza:~]$ docker version
Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:48:57 2018
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:16:44 2018
OS/Arch: linux/amd64
Experimental: false

[reza:~]$ minikube version
minikube version: v0.33.1

‍‍‍‍```

[tstromberg]

Next time someone runs into this, please attach the output from

minikube start
minikube logs
minikube ssh 'docker logs $(docker ps -a -f name=k8s_kube-api --format={{.ID}})'

Also, if you are seeing this issue on something other than the vmware/vbox combination, please open a new bug as I suspect you may be having a separate issue with similar symptoms.

Thanks!