minikube mount not working properly on macOS with xhyve

[wescarr]

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

Minikube version: v0.17.1

Environment:

• OS: MacOS Sierra 10.12.2
• VM Driver: xhyve
• ISO version: minikube-v1.0.7.iso
• Install tools: Installed via https://github.com/kubernetes/minikube/releases
• Others:

What happened:
While the /mount-9p directory is created inside the VM, it does not contain any of the host files. Running minikube ssh -- ls -lah /mount-9p yeilds:

total 0
drwxrwxr-x    2 root     root          40 Mar  3 23:57 .
drwxr-xr-x   18 root     root         500 Mar  3 23:57 ..

What you expected to happen:
Running minikube ssh -- ls /mount-9p to give the same output as ls /k8smount.

How to reproduce it (as minimally and precisely as possible):

• sudo mkdir /k8smount # Create host mount dir
• sudo touch /k8smount/hello.txt
• minikube start --vm-driver=xhyve
• minikube mount /k8smount &
• minikube ssh -- ls -lah /mount-9p

Anything else do we need to know:
Discussed in slack #minikube channel and confirmed by @imathews. cc @aaron-prindle

• I tried changing the permissions on the host dir as well: chmod -R 777 /k8mount to no avail.
• Creating files in either the VM or the host had no effect on the other.
• I was able to sudo rm -rf /mount-9p in the VM while running minikube mount with no error. Once the dir was removed, it was not recreated.

[aaron-prindle]

Can you post the output of running:
minikube mount --v=3 /k8smount

[wescarr]

Sure this is all I get

$ minikube mount --v=3 /k8smount
Mounting /k8smount into /mount-9p on the minikubeVM
This daemon process needs to stay alive for the mount to still be accessible...

[henridf]

Similar issue here.

Matches all of @wescarr's report except for the OS X version which is El Capitan 10.11.6

[aaron-prindle]

Can you try mounting a non-root owned directory with the mount command? I think this might be what is causing the issue.

[fennb]

I believe I've found the cause of this, it appears (at least in my case) that the minikube command doesn't detect/pass through the correct IP address for the server:

$ minikube mount -v 15 --alsologtostderr /Users
I0316 17:45:08.523645   56932 notify.go:112] Checking for updates...
Mounting /Users into /mount-9p on the minikubeVM
This daemon process needs to stay alive for the mount to still be accessible...
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:55245
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/fenn/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /Users/fenn/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:

sudo mkdir /mount-9p;
sudo mount -t 9p -o trans=tcp -o port=5640 10.0.2.2 /mount-9p;
sudo chmod 775 /mount-9p;

Then eventually:

err     : exit status 1
output  : mkdir: can't create directory '/mount-9p': File exists
mount: mount 10.0.2.2 on /mount-9p failed: Connection timed out
chmod: /mount-9p: Input/output error

This gives the appearance once you're inside the machine of the mount "working" but in fact, only the sudo mkdir has succeeded and there is no mount present.

Note the IP of 10.0.2.2. Now, if I ssh into the machine:

$ minikube ssh
$ ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:93:F9:81:5C
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:93ff:fef9:815c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1303 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1336 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:118521 (115.7 KiB)  TX bytes:374077 (365.3 KiB)

eth0      Link encap:Ethernet  HWaddr 9E:55:33:66:CF:32
          inet addr:192.168.64.3  Bcast:192.168.64.255  Mask:255.255.255.0
          inet6 addr: fe80::9c55:33ff:fe66:cf32/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:79110 errors:0 dropped:0 overruns:0 frame:52
          TX packets:15473 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:109613179 (104.5 MiB)  TX bytes:1213786 (1.1 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:23251 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23251 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:10691922 (10.1 MiB)  TX bytes:10691922 (10.1 MiB)

veth490f180 Link encap:Ethernet  HWaddr 86:48:5E:83:79:FE
          inet6 addr: fe80::8448:5eff:fe83:79fe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:222 errors:0 dropped:0 overruns:0 frame:0
          TX packets:268 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:44112 (43.0 KiB)  TX bytes:24846 (24.2 KiB)

vethf5d146c Link encap:Ethernet  HWaddr CA:AC:D8:39:36:5B
          inet6 addr: fe80::c8ac:d8ff:fe39:365b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1081 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1118 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:92651 (90.4 KiB)  TX bytes:353295 (345.0 KiB)

The eth0 address of the minikube vm is 192.168.64.3.

If I run the mount command manually within the VM using the determined IP, it succeeds:

$ sudo mount -t 9p -o trans=tcp -o port=5640 192.168.64.1 /mount-9p
$ mount | grep mount-9p
192.168.64.1 on /mount-9p type 9p (rw,relatime,sync,dirsync,trans=tcp,port=5640)
192.168.64.1 on /mount-9p type 9p (rw,relatime,sync,dirsync,trans=tcp,port=5640)
$ ls /mount-9p | wc -l
19

Looks like it's just the IP/subnet detection from the minikube mount command that has an issue.

[aaron-prindle]

I was under the impression that the host ip from within the VM to the host was a static 10.0.2.2 for xhyve. It is strange as this value appears to be working for some users but not for others. We should add better validation/error messages for the mount command and make it so that the mount ip for xhyve is generated by looking at the host net interfaces similar to Windows hyperv:
https://github.com/kubernetes/minikube/blob/master/pkg/minikube/cluster/cluster_windows.go#L60-L79

Thanks for investigating this!

[fennb]

@aaron-prindle my pleasure!

Do you happen to know if there's a way to specify the subnet that xhyve sets up for minikube VM?

I didn't do anything special doing set up, but ended up with 192.168.64.0/24 one way or another.

[alexef]

Also hitting this, minikube v0.17.1 + xhyve.

[aaron-prindle]

Closing. This should be fixed now with #1293 and will be present in minikube v0.18.0. It seems that xhyve has 192.168.64.1 mapped back to host from vmnet. From:
https://github.com/zchee/docker-machine-driver-xhyve#does-not-clean-up-the-vmnet-when-remove-a-vm
Note that vmnet.framework shared net address range is ~ 192.168.64.255...