Minikube: hyperv: minikube stuck at "Waiting for SSH access ..."

[MichelMelhem]

When i start my minikube cluster i have these logs,

$ minikube start
o   minikube v0.34.1 on windows (amd64)
i   Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
:   Re-using the currently running hyperv VM for "minikube" ...
:   Waiting for SSH access ...

And it stuck at waiting for SSH access.
I use hyper V and when i connect to the vm the minikube cluster start successfully but in my console don't show it.
If i kill the command and i try to run minikube delete i have these logs
```
$ minikube delete

• Powering off "minikube" via SSH ...
  ```
  And the command stuck at "Powering off "minikube" via SSH ..."

Im using:
Windows v1809
minikube v0.34.1

Thank you in advance for your help !

[tstromberg]

Do you mind including the output of:

minikube start --alsologtostderr -v=8

Also, the ssh should timeout after 5-10 minutes. Do you mind letting it timeout and outputting the resulting error message here?

My suspicion is that the networking configuration in HyperV isn't routing the network packets properly. The documentation mentions a workaround that might work for you:

https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperv-driver

Please let me know if the workaround helps. Thank you for your bug report!

[MichelMelhem]

This is the outpout of minikube start --alsologtostderr -v=8
``` minikube start --alsologtostderr -v=8
I0222 19:36:30.141247 6412 notify.go:121] Checking for updates...
o minikube v0.34.1 on windows (amd64)
I0222 19:36:30.171247 6412 start.go:579] Saving config:
{
"MachineConfig": {
"MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v0.34.0.iso",
"Memory": 2048,
"CPUs": 2,
"DiskSize": 20000,
"VMDriver": "hyperv",
"ContainerRuntime": "docker",
"HyperkitVpnKitSock": "",
"HyperkitVSockPorts": [],
"XhyveDiskDriver": "ahci-hd",
"DockerEnv": null,
"InsecureRegistry": null,
"RegistryMirror": null,
"HostOnlyCIDR": "192.168.99.1/24",
"HypervVirtualSwitch": "MinikubeNAT",
"KvmNetwork": "default",
"DockerOpt": null,
"DisableDriverMounts": false,
"NFSShare": [],
"NFSSharesRoot": "/nfsshares",
"UUID": "",
"GPU": false
},
"KubernetesConfig": {
"KubernetesVersion": "v1.13.3",
"NodeIP": "",
"NodePort": 8443,
"NodeName": "minikube",
"APIServerName": "minikubeCA",
"APIServerNames": null,
"APIServerIPs": null,
"DNSDomain": "cluster.local",
"ContainerRuntime": "docker",
"CRISocket": "",
"NetworkPlugin": "",
"FeatureGates": "",
"ServiceCIDR": "10.96.0.0/12",
"ExtraOptions": null,
"ShouldLoadCachedImages": false,
"EnableDefaultCNI": false
}
}
I0222 19:36:30.173248 6412 cluster.go:75] Skipping create...Using existing machine configuration
i Tip: Use 'minikube start -p ' to create a new cluster, or 'minikube delete' to delete this one.
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
I0222 19:36:30.558261 6412 cluster.go:94] Machine state: Running
: Re-using the currently running hyperv VM for "minikube" ...
I0222 19:36:30.558261 6412 cluster.go:112] engine options: &{ArbitraryFlags:[] DNS:[] GraphDir: Env:[] Ipv6:false InsecureRegistry:[10.96.0.0/12] Labels:[] LogLevel: StorageDriver: SelinuxEnabled:false TLSVerify:false RegistryMirror:[] InstallURL:}
: Waiting for SSH access ...
Waiting for SSH to be available...
Getting to WaitForSSH function...
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Hyper-V\Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : 192.168.1.46

[stderr =====>] :
Using SSH client type: native
&{{{ 0 [] [] []} docker [0x8359b0] 0x835980 [] 0s} 192.168.1.46 22 }
About to run SSH command:
exit 0
Error dialing TCP: dial tcp 192.168.1.46:22: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
```
And i don't think is a network switchproblem because my network switch is well configured

[MichelMelhem]

And when i do minikube ip i successfully get an ip,

[RomanIakovlev]

I experience pretty much same error, and logs are almost identical (modulo ip address value). My minikube version is same, 0.34.1, and Windows version is 1803.

[RomanIakovlev]

I believe I found the workaround/solution. I had a flashback about similar situation I had with Docker some time ago. The root cause seems to be Windows Firewall, which by default treats Hyper-V virtual switches as an untrusted network. The solution is to switch its category to private, with the following PowerShell command (you must run PowerShell with admin rights), presuming your virtual switch for minikube is called "minikube":

Set-NetConnectionProfile -interfacealias "vEthernet (minikube)" -NetworkCategory Private

You probably have to repeat it after each Windows update, because this change doesn't seem to persist after such updates.

P.S. I wasn't able to make things work in general due to another error further down the road (I should probably file another issue for it), but at least I've unblocked this general connectivity issue between host and VM.

Hope this helps.

[MichelMelhem]

Thank you for your help !

[MichelMelhem]

But i finally decide do use the kubernetes inegration bundle with the docker desktop app because it it is more simple to use than minikube and it has less issues

[matthiasg]

@The-Developeur what is the integration bundle ?

[ravishankarbanka]

I am facing the same problem, with two exceptions ,

1. it seems like mine uses ipv6
2. I get to see below error
   "Error configuring auth on host: OS type not recognized"
   I checked I am not able to ping the ipv6 address returned by "minikube ip" command

not sure how to fix it. any help or suggestion?

[MichelMelhem]

@matthiasg This bundle

[kuchaguangjie]

I got the same issue, it's solved by stop the vm in virtulbox, and then run minikube start again, then the error is gone.

BTW:

• run minikube stop directly, won't work, because it will stuck too.
• kill minikube processes also doesn't fix it.

[EveraldB]

With Hyper-V
It was the VM switch configuration error
The can only be one external VM switch associated with active NIC card
Power shell scripts showed me the error

PS C:> Get-NetAdapter
$net = Get-NetAdapter -Name 'Ethernet'
New-VMSwitch -Name "External VM Switch" -AllowManagementOS $True -NetAdapterName $net.Name

[efengx]

The same problem occurs on mac computers:（Close after minikube restarts）

fengxiangdeMacBook-Pro% minikube start --logtostderr --v=8\
 --memory=4096 --cpus=4\
 --vm-driver hyperkit\
 --extra-config=apiserver.authorization-mode=RBAC\
 --registry-mirror=https://registry.docker-cn.com\
 --docker-env NO_PROXY=localhost,127.0.0.1,192.168.99.0/24,192.168.64.0/24\
 --docker-env HTTP_PROXY=http://192.168.0.104:1087\
 --docker-env HTTPS_PROXY=http://192.168.0.104:1087
I0327 10:33:41.885774    5845 notify.go:121] Checking for updates...
W0327 10:33:52.337649    5845 notify.go:55] Error getting json from minikube version url: error with http GET for endpoint https://storage.googleapis.com/minikube/releases.json: Get https://storage.googleapis.com/minikube/releases.json: net/http: TLS handshake timeout
😄  minikube v0.35.0 on darwin (amd64)
I0327 10:33:52.339206    5845 start.go:582] Saving config:
{
    "MachineConfig": {
        "MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v0.35.0.iso",
        "Memory": 4096,
        "CPUs": 4,
        "DiskSize": 20000,
        "VMDriver": "hyperkit",
        "ContainerRuntime": "docker",
        "HyperkitVpnKitSock": "",
        "HyperkitVSockPorts": [],
        "XhyveDiskDriver": "ahci-hd",
        "DockerEnv": [
            "NO_PROXY=localhost,127.0.0.1,192.168.99.0/24,192.168.64.0/24",
            "HTTP_PROXY=http://192.168.0.104:1087",
            "HTTPS_PROXY=http://192.168.0.104:1087"
        ],
        "InsecureRegistry": null,
        "RegistryMirror": [
            "https://registry.docker-cn.com"
        ],
        "HostOnlyCIDR": "192.168.99.1/24",
        "HypervVirtualSwitch": "",
        "KvmNetwork": "default",
        "DockerOpt": null,
        "DisableDriverMounts": false,
        "NFSShare": [],
        "NFSSharesRoot": "/nfsshares",
        "UUID": "",
        "GPU": false,
        "NoVTXCheck": false
    },
    "KubernetesConfig": {
        "KubernetesVersion": "v1.13.4",
        "NodeIP": "",
        "NodePort": 8443,
        "NodeName": "minikube",
        "APIServerName": "minikubeCA",
        "APIServerNames": null,
        "APIServerIPs": null,
        "DNSDomain": "cluster.local",
        "ContainerRuntime": "docker",
        "CRISocket": "",
        "NetworkPlugin": "",
        "FeatureGates": "",
        "ServiceCIDR": "10.96.0.0/12",
        "ExtraOptions": [
            {
                "Component": "apiserver",
                "Key": "authorization-mode",
                "Value": "RBAC"
            }
        ],
        "ShouldLoadCachedImages": false,
        "EnableDefaultCNI": false
    }
}
I0327 10:33:52.340334    5845 cluster.go:75] Skipping create...Using existing machine configuration
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:52292
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .DriverName
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
(minikube) Calling .GetState
I0327 10:33:52.361674    5845 cluster.go:94] Machine state:  Stopped
(minikube) Calling .DriverName
🔄  Restarting existing hyperkit VM for "minikube" ...
(minikube) Calling .Start
(minikube) Using UUID f9da217a-4e1f-11e9-8026-a0999b193f0d
(minikube) Generated MAC 82:aa:49:4:a1:38
(minikube) Starting with cmdline: loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube
(minikube) Calling .GetConfigRaw
I0327 10:33:52.626659    5845 cluster.go:112] engine options: &{ArbitraryFlags:[] DNS:[] GraphDir: Env:[NO_PROXY=localhost,127.0.0.1,192.168.99.0/24,192.168.64.0/24 HTTP_PROXY=http://192.168.0.104:1087 HTTPS_PROXY=http://192.168.0.104:1087] Ipv6:false InsecureRegistry:[10.96.0.0/12] Labels:[] LogLevel: StorageDriver: SelinuxEnabled:false TLSVerify:false RegistryMirror:[https://registry.docker-cn.com] InstallURL:}
⌛  Waiting for SSH access ...
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x143d0e0] 0x143d0b0  [] 0s} 192.168.64.9 22 <nil> <nil>}
About to run SSH command:
exit 0
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out
Error dialing TCP: dial tcp 192.168.64.9:22: connect: operation timed out

It seems that vm didn't start, I can't find this vm

[efengx]

After performing the naming, you can start it. But this operation looks stupid

rm -rf ~/.minikube/machines/minikube/hyperkit.pid

Heavy scene:
Instead of shutting down the minikube via minikube stop, it forces the computer to power off. Then the error occurs in minikube start.

[cwhsu1984]

Why is this issue even closed? I try to install minkube on Windows 10 today, and I'm still stuck here after 8 hours.

First, it's this one.

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8359b0] 0x835980  [] 0s} 192.168.1.104 22 <nil> <nil>}
About to run SSH command:
exit 0
Error dialing TCP: dial tcp 192.168.1.104:22: connectex: No connection could be made because the target machine actively refused it.

I read a lot of articles, and try to stop and start adaper, try to change Virtaul Swtich Connection type from External network to private, try to change External network from Wi-Fi to ethernet.

[stderr =====>] :
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x8359b0] 0x835980  [] 0s} fe80::215:5dff:fe73:bb03 22 <nil> <nil>}
About to run SSH command:
exit 0
Error dialing TCP: dial tcp [fe80::215:5dff:fe73:bb03]:22: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

None of those work. Then I try to delete minikube to try a new minikube VM, then I cannot stop it. I cannot delete it. I cannot even shut it down from Hyper-V Manager. I have to reboot into Safe-Mode just to delete C:\Users\USER\.minikube to go back and create minikube from start AGAIN.

I have been using docker on Ubuntu for years, and I couldn't imagine how windows users are going to use something like this. After trying to run minikube on windows for 8 hours, I am just very frustrated.

[mentos1386]

@tstromberg Can this be reopened as original issue is still a problem and switching to Docker for Mac/Windows isn't really a proper solution.

For me on macos with hyperkit doing minikube delete works. And then minikube start works, as it creates new VM. But it's not a nice workaround.

[oscar6echo]

On macOS with minikube version 1.0.0, upon a hanging

Waiting for SSH access ...

it was enough to do

^C                           # interrupt
minikube delete     # delete minikube from virtualbox and detele cluster
minikube start

to get a working minikube.

[repetty]

did all the hard things first but a restart after
having used CiscoVPN a couple weeks ago is
what cleared the minikube startup problem I
was having ("Waiting for SSH access ...").

A Mac host restart cleared the problem for me and this seems like a significant clue to the problem... Maybe my Mac's networking state was in an unexpected configuration as a result of running CiscoVPN. Other's have reported success after doing upgrades, downgrades, reinstalls, et cetera, but I'm wondering if they did not realize that they did other things that might have also had an effect on curing their problem... like rebooting or sleeping (I have done neither for almost a month).

[yinzara]

Requiring a minikube delete everytime there is a minikube stop is not a valid solution. What's causing this?

[robymontyz]

I had the same issue with hyperkit. I resolved the problem today upgrading to the latest version of minikube (1.2.0) and installing the corresponding version of hyperkit driver from here.

• To update minikube:

curl -Lo minikube https://github.com/kubernetes/minikube/releases/download/v1.2.0/<minikube-for-your-OS> && chmod +x minikube && sudo cp minikube /usr/local/bin/ && rm minikube

Remember to replace <minikube-for-your-OS> with the corresponding version of your operating system. Check the link provided above with all the releases for details.

• To update hyperkit driver:

curl -LO https://github.com/kubernetes/minikube/releases/download/v1.2.0/docker-machine-driver-hyperkit && sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/

[conterio]

Any update to a permanent solution?

[seamc]

(Win10, minikube 1.2.0) I had been struggling with this for a little bit, and minikube holds fast through reboots and hyper-v service restarts after I disabled all Virtual Switch Extensions (particularly Microsoft NDIS Capture) on the External Switch that I use for minikube. My virtual switch ("k8s") is the only external switch managed by hyper-v; everything else is private or internal.

Specifically within the minikube VM settings in hyper-v, all of the Network Adapter settings are whatever default settings applied by minikube start or hyper-v.

Windows Firewall is disabled.

minikube start --memory 4096 --vm-driver=hyperv --hyperv-virtual-switch="k8s" --alsologtostderr -v=8

[Fazered]

For me the issue was using assigning a network adapter in my laptop dock and then moving dock. To keep it consistent I now just use the wifi for my external switch and connect both when I am using minikube.

[yinzara]

I have seen that when a USB network adapter (like one found in a dock) is present, minikube fails to start after being stopped with the "Waiting for SSH..." message. Using the new "--no-native-ssh" flag caused the problem to disappear (though there is debate about the flag and it probably will have its name changed before the PR is accepted).

[ukiahsmith]

I experience this with:

macOS: High Sierra 10.13.6
minikube version: v1.2.0
using: --vm-driver hyperkit
hyperkit: v0.20180403-17-g3e954c

I was able to resolve it with efengx's solution:

rm -rf ~/.minikube/machines/minikube/hyperkit.pid

[rainfd]

I have the same experience with efengx and UkiahSmith.
Just delete the file hyperkit.pid

My environment:
macOS: Catalina 10.15
minikube: v1.2.0
hyperkit: v0.20190201-11-gc0dd46