K3s: macOS Support

[seanabraham]

Describe the bug
As far as I can tell, the k3s x86_64 binary does not run on macOS. When invoking the binary, I see the following:

$ ./k3s
bash: ./k3s: cannot execute binary file

I've tried to build the binary on macOS itself with little luck (though I could be missing something, I'm not super familiar with go compilation):

❯ GOOS=darwin GOARCH=amd64 go build -o k3s
# github.com/rancher/k3s/vendor/github.com/coreos/flannel/pkg/ip
../../personal/go/src/github.com/rancher/k3s/vendor/github.com/coreos/flannel/pkg/ip/iface.go:124:17: undefined: netlink.RouteGet
../../personal/go/src/github.com/rancher/k3s/vendor/github.com/coreos/flannel/pkg/ip/iface.go:136:47: undefined: netlink.FAMILY_V4
../../personal/go/src/github.com/rancher/k3s/vendor/github.com/coreos/flannel/pkg/ip/tun.go:56:18: undefined: syscall.IFF_TUN
../../personal/go/src/github.com/rancher/k3s/vendor/github.com/coreos/flannel/pkg/ip/tun.go:56:36: undefined: syscall.IFF_NO_PI
../../personal/go/src/github.com/rancher/k3s/vendor/github.com/coreos/flannel/pkg/ip/tun.go:58:29: undefined: syscall.TUNSETIFF
# github.com/rancher/k3s/pkg/agent/containerd
../../personal/go/src/github.com/rancher/k3s/pkg/agent/containerd/containerd.go:86:4: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr

I haven't looked into each of the errors here, but usage of Pdeathsig for example, seems to only be available on Linux per https://golang.org/pkg/syscall/#SysProcAttr

To Reproduce
Steps to reproduce the behavior:
Run k3s on macOS or try to build on macOS.

Expected behavior
Binary runs successfully

[unixfox]

Is native kubernetes (k8s) working on MacOS without using a VM? Because I think it's expected to fail on MacOS if kubernetes itself isn't compatible with macos.

[ibuildthecloud]

k3s is Kubernetes which runs on Linux only (well Windows too, but lets not go there yet). So the only way you could run k3s on macOS is if we made a wrapper so that k3s binary launched a VM on your mac, similar to how docker for mac works. We are looking at this use case, but quite a bit of work.

We will do our best to put together some docs on the best way to use k3s on macOS. Either using docker for mac, vagrant, or something similar.

[seanabraham]

Right that makes perfect sense, thanks @unixfox & @ibuildthecloud — documentation or any convenience utilities to configure the appropriate environment would be much appreciated.

[tedmiston]

@ibuildthecloud I ended up here after trying to run the binary on macOS as well. I figured it may or may not be supported but couldn't find docs about macOS specifically.

Is there an issue we can follow for k3s docs on macOS using e.g, the setup you described?

[vitobotta]

Oh :( I was trying to use K3D for development/testing on my Mac and I see now why the binary doesn't work :(

[giminni]

I am testing k3d and IMHO is a good alternative to minikube and friends. It can mimic a big k3s cluster with a lot of workers.

If you look closely Erik implemented in v0.7.0-rc2 ctr on k3s to include even a registry for images. Ctr is working on the containerd side for image inclusion.

I don't think we need a MacOSX k3s implementation. Let the core team focus more on other things like HA, stability and k3d integration.

[derberg]

I recommend multipass to quickly setup ubuntu vm on mac and run k3s
This step by step doc helped me out
https://medium.com/@zhimin.wen/running-k3s-with-multipass-on-mac-fbd559966f7c

[sgandon]

Well we are thinking of using K3s at our customer premises but we don't want to contrain them into using it on linux. And I am afraid K3d adds some ovehead in term of complexity and perf that may not be suitable for production workload.
Since the code is in Go, it's it possible to build it for mac or windows ? or does it uses native linux libs?

[chickenandpork]

The code that k3s uses is written to use Linux sysctl() methodology: rather than generic actions such as "get the IPv4 IP" with a system-specific library describing how to do this (which would be via sysctl() calls on linux) the code is written to open sysctls (https://github.com/containernetworking/plugins/blob/5c512194ebea90cda344c3ca371f81c6ac2420a3/pkg/utils/sysctl/sysctl_linux.go#L48,L56) .. I would assume this was done simply because it worked, and their target audience was linux. In current code, for non-linux architecture, there'd need to be a sysctl_macos.go or sysctl_windows.go that would need to translate /proc/sys/<namedThing> into the Mac or Windows equiv.

I guess it would make sense to make a generic function that in turn calls linux-specific sysctl() calls, or the macOS way, or the windows way of doing a thing, but that would require some more abstract function calls layering between current code and the sysctl() stuff. Not trivial work, and harder because container networking would need to accept the changes, not k3s.