Minikube: Make xhyve default for OSX

[dlorenc]

After https://github.com/kubernetes/minikube/pull/248 lands, we'll have these remaining tasks for xhyve support are:

• Figure out which version of docker-machine-driver-xhyve to use
• Figure out which version of libxhyve to use
• Bundle these into minikube
• Make it default

[jimmidyson]

FYI I've made xhyve, kvm, hyperv default for OS X, Linux, Windows respectively on a fork (https://github.com/jimmidyson/minishift) & all works great. Some issues related to setting up binaries but nothing any different from having VirtualBox installed & configured...

[sandinmyjoints]

Is it possible to pass flags to docker-machine-driver-xhyve?

[dlorenc]

Not right now, which flags are you trying to pass?

On Thu, Aug 11, 2016, 4:49 PM William Bert [email protected] wrote:

Is it possible to pass flags
https://github.com/zchee/docker-machine-driver-xhyve#available-flags to
docker-machine-driver-xhyve?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/minikube/issues/270#issuecomment-239327088,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABopNtq9IU0We2fNXgoEt8muUyBiPeseks5qe7T8gaJpZM4JEc4D
.

[sandinmyjoints]

I'd beef up xhyve a bit, something like: --xhyve-cpu-count=2 --xhyve-memory-size=2048.

[dlorenc]

Ah, you can just use our standard CPU and memory flags, those will go to the driver.

[sandinmyjoints]

Great, thanks!

[justechn]

Do you guys enable --xhyve-experimental-nfs-share by default? If not, is there a way to enable it?

[r2d4]

@justechn no, we don't expose it or enable it. @aaron-prindle is working on a cross platform nfs solution tracked by #1033.

In the meantime, if you really want to set up an nfs mount on OSX, you can follow some of the instructions here https://github.com/kubernetes/minikube/issues/2#issuecomment-272070149

[justechn]

I recommend against making xhyve the default driver. For me trying to get xhyve to work reliably has been a big pain in the rear. There have been numerous fixes put into xhyve, but getting zchee to cut new releases is proving difficult. There is also a networking issue that is blocking some, including me from being able to use it https://github.com/zchee/docker-machine-driver-xhyve/issues/107. Virtualbox may not be as performant, but it works a lot more reliably, and for a default driver I think that is the most important.

Personally I would rather see more effort put into the no driver options. I think running minikube in local docker is a much better idea

[copumpkin]

@justechn how do you mean local docker? Unless I'm misunderstanding something, the "none" driver only really makes sense on Linux (and possibly Illumos), which can natively run Linux containers with namespace/cgroup support built into the kernel. macOS has no such support and as such needs some sort of VM to run Linux.

[justechn]

I was hoping that was not the case. Oh well...

[copumpkin]

Yeah, unfortunately containers are basically a special way to run a process (tree) on linux, where you can say "I want this process I'm launching to have a distinct view of processes, filesystems, network devices, users, and some assorted other stuff; also I want to control its lifecycle/resource consumption". The first half of that is namespaces and the second half of that is cgroups, and macOS has neither of those mechanisms. It does have a distinct kernel sandboxing mechanism that gives you some of the benefits, but given that all container mechanisms out there assume Linux, if you tried to download and run a Docker container (even isolated using the sandbox mechanism), the macOS kernel would just say "I don't know how to run any of the binaries in here".

[jgiles]

Question: What are the benefits of xhyve over VirtualBox?

The main comparison I've found is here: https://thepracticalsysadmin.com/xhyve-vs-vbox-driver-benchmarks-for-docker-machine/

It's a bit dated at this point (Feb 2016), but suggests VirtualBox has comparable and potentially better performance. Do we have more recent/higher quality evidence that xhyve yields better performance, reliability, etc? Or is switching to xhyve just about simplifying installation/maintenance?

For context: I'm trying to reduce the runtime of a bunch of tests executing in a minikube environment, and am considering swapping from the vbox to the xhyve driver, but haven't found any benchmarks suggesting it will actually improve performance.

[alanbrent]

@jgiles The TLDR is that xhyve-based minikube environments are better citizens in macOS. As an example: In my experience, VirtualBox takes ALL the memory you alot a VM and never gives it back. That's not the case w/ xhyve-based environments.

[coderanger]

This should probably be updated to say "Make hyperkit the default for OSX" since that is the current recommendation.

[jgiles]

Also, as a followup to https://github.com/kubernetes/minikube/issues/270#issuecomment-328937857 above: I tried xhyve for a while, and at least in my case there was a very noticeable loss of performance relative to VirtualBox. Before switching the default, it would be good to get some performance metrics for minikube workloads on each of the drivers.
Performance doesn't need to be a dominant factor in selecting the default driver, but understanding and communicating the tradeoffs in driver choice would be very useful.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[coderanger]

This is still something that needs to be reviewed.

[itaysk]

What’s the current recommendation for best experience (I don’t mind minor stability issues)? vbox/xhyve/hyperkit?