K3s: K3s OS

[bencompton]

In the K3s meetup video, the dedicated K3s OS briefly mentioned is an exciting idea. I am guessing this would mean that K3s would be running as the first process like Docker does in RancherOS, and that system containers would run in K3s (perhaps in a specific namespace)?

[iluminae]

Cool idea. I created my own initramfs that is hosted from my PXE server that runs k3s server as pid 1. The initramfs only includes busybox and the installed components of nfs-utils and open-iscsi from Alpine, so I can mount NFS and iSCSI. /var/lib/rancher is stored on a iSCSI block.

[bencompton]

@iluminae - Nice! Seems you could run a busybox container in K3s instead, though not sure that would get you much other than aesthetics.

Here are some other wish list items that come to mind for a custom K3s OS:

• SSH disabled by default in favor of centralized, declarative configuration. What if there were a custom Kubernetes API extension for managing the OS itself (modeled after cloud-config in RancherOS)? Imagine being able to pass in a manifest to initialize each node and then centrally roll out OS configuration updates via kubectl, including OS patches!

• Automates most of the common operational requirements, like backup / restore

• Production hardened out of the box, perhaps including SELinux (or similar MAC solution) with a sensible default policy. This could be difficult to get right without being too opinionated and restrictive, but it might be possible to have an array of high-level configuration options that would cover most use cases. In general, less control for stronger security seems like an equitable tradeoff.

[discordianfish]

I guess rancher would recommend rancheros to be the place to run k3s? See #173

[bencompton]

I actually saw #173 the other day, and bundling K3s in RancherOS would be a nice time saver. I've never used RancherOS other than messing around with it, but I love the idea. RancherOS is my favorite prospect for running K3s so far.

However, it does seem like overkill to run Docker when the goal is to run K3s. K3s + a kernel is essentially a whole OS in itself. Also, I'm sure there would be a lot of opportunities to improve the experience with a dedicated, specialized OS, such as the items I mentioned above.

[discordianfish]

I'm searching for a option as well. There is balenaOS and hypriot as well. Personally though I'd prefer something very minimal, but with all the firmware stuff needed for rpi3.

[spmason]

Looks like rancher just released this: https://rancher.com/blog/2019/announcing-k3os-kubernetes-operating-system/