Microk8s: Windows container support?

[biiiipy]

Is it possible to add Windows worker nodes to microk8s cluster? Haven't seen this topology mentioned anywhere.
So basically:
microk8s clusters in WSL2 and plain windows worker nodes with windows containers (without running the microk8s on the windows worker nodes, which would be linux containers).

[balchua]

Joining non microk8s nodes to a microk8s cluster you mean?

[biiiipy]

Yes, non-microk8s nodes (plain windows worker nodes) to microk8s clusters running on WSL2.
The goal is to have microk8s cluster running in WSL2, BUT run windows containers (on Windows Server 2019).

[balchua]

From what i can tell, not possible.

[joedborg]

Hi @biiiipy, to get the Linux workers (i.e. MicroK8s) on Windows, we recommend using the Windows installer found here: https://microk8s.io/docs/install-alternatives#heading--windows.

Getting the Windows workers running would be a different challenge, but (I think), if you could get MicroK8s running with the installer, you could enroll manually installed Windows Kubelets; in theory at least. There is no precedence for this, but I'd be interested if you get it working.

[organom]

I've already deployed a native kubernetes cluster working with both linux and windows nodes, but its a real pain, lots of configuration and a huge mess with the windows nodes having to proxy the traffic via the linux nodes.

I recently discovered microk8s and really enjoyed installing a PoC cluster with it, its stable, fast and easy to use. But having windows containers running in the same k8s cluster is, unfortunately, also a need in the project.

From the microk8s side, we have no access to the kubeadm so we can't do kubeadm token list or kubeadm token create $(kubeadm token generate). microk8s add-node will generate something like microk8s join 192.168.1.154:25000/be0f5f9e286f7e81b1b62e45719048a2
From the native node side, we would generally run something like kubeadm join 192.168.1.146:6443 --token 9a1qzb.ggdku08tuu2bzv8h --discovery-token-ca-cert-hash sha256:682dc1573fc3b6ca71f99f19979c99b07b7190248f2ff72a5b6865ad265489ad to join the node to the cluster.

@joedborg how would you enroll a manual windows kubernetes? I tried to search in the documentation but couldn't find anything :'(
Any idea how we could get the needed values out of microk8s in order to try to join the native node with them ?

Edit 16/10/2020 11:31: I found that we can get the certificate from microk8s with microk8s config in the field certificate-authority-data that is base64. So i just decoded it copied it to a file ca.crt and run openssl x509 -in ca.crt -noout -pubkey | openssl rsa -pubin -outform DER 2>/dev/null | sha256sum | cut -d' ' -f1 this theoretically gets me the sha256 value of the cert to append to --discovery-token-ca-cert-hash sha256: . So I'm just missing the token, since the one generated by the add-node isn't in the format of the native kubeadm token.

[ktsakalozos]

Hi @organom,

all certs are in /var/snap/microk8s/current/certs and credentials (eg tokens etc) are in /var/snap/microk8s/current/credentials. In case you need to configure any services you will find the services arguments in /var/snap/microk8s/current/args .

Are you following any guide/doc I could look at? How do you install the windows node?

[biiiipy]

@organom great to see someone else in similar situation :)
Also, did I understand correctly, that we don't need dedicated Linux nodes with microk8s on Windows, cause microk8s runs them in Hyper-V Ubuntu VMs (I only tried to go WSL2+Microk8s route)?
So the missing piece is getting windows native nodes (for native windows containers) to join cluster running in microk8s? If that's the case, then this makes microk8s the best solution for Windows centric environments, by a huge margin, IF we can get windows containers working, of course...

[organom]

@biiiipy yes, that is my understanding of it also. On a native kubernetes, you setup the master, Linux nodes and then the windows nodes. Linux nodes are still mandatory because the windows machines use them to proxy the traffic.
On microK8s, there is no need for master (each installation is master and node), so a single machine would suffice. The windows, macos or linux installation in case of microk8s is always the same, a linux node/master running. Would be great if microk8s would add a flag to also boot a windows native node, but since this is probably not an easy feat, my thought was to keep the windows nodes as native kubernetes installation, add the required proxy updates to microk8s and try to somehow join them to the microk8s cluster

@ktsakalozos I followed the native kubernetes installation guide https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/ and https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/ .

As a first step I would be happy to at least be able to join the native windows node into a microk8s cluster, but the token difference (9a1qzb.ggdku08tuu2bzv8h expected from native instead of be0f5f9e286f7e81b1b62e45719048a2 that microk8s gives) is preventing me from testing it, after that there is still the question of the network CNI flannel specific windows configurations (that in the case of microk8s would need to be calico), but one fight at a time ;)

[zejji]

+1 for this. We have legacy .NET Framework containers which can only be run on a native Windows worker, so it would be very helpful to be able to easily spin up such a worker using microk8s!

[joedborg]

We've just finished the documentation for this:

https://discuss.kubernetes.io/t/add-a-windows-worker-node-to-microk8s/13782

[organom]

We've just finished the documentation for this:

https://discuss.kubernetes.io/t/add-a-windows-worker-node-to-microk8s/13782

@joedborg The document looks awesome, but it misses (I believe) the part of the joining the windows node to the microk8s node. How do you do it?

Thanks for the help

[joedborg]

@organom When you start the kubelet service (towards the end), it connects by reading the kubeconfig file you wrote to the Windows node.

[organom]

@joedborg I see, didn't knew kubelet would accept it like that :) I will give it a try next week.
Thank you again for writing this up and letting us know ;)

[joedborg]

@organom yeah, the kubeconfig contains all you need to reach and authenticate with the control plane.

No problem, I'll close this issue. Please feel free to open a new one if you come across anything specific to the documentation.

[organom]

Just a fast update from my side. Worked like a charm.
Following the tutorial I was able to have the windows node joined to the microk8s cluster and run a simple windows test pod (powershell curl of google.com).

Thank you again for making the tutorial available

[matt-psaltis]

Trying to work out if this can be done entirely from a Windows 10 (pro/ent) host. microk8s on WSL2 (ubuntu)
What are the options for the Windows Worker?

• Hyper-V Server 2019 VM?
• Can this work with the Win 10 host directly? I.E. it becomes the worker?

How are other people configuring this?

[organom]

Trying to work out if this can be done entirely from a Windows 10 (pro/ent) host. microk8s on WSL2 (ubuntu)
What are the options for the Windows Worker?

* Hyper-V Server 2019 VM?

* Can this work with the Win 10 host directly? I.E. it becomes the worker?

How are other people configuring this?

There are 2 important concepts to separate here:

• one thing is running microk8s in windows
• other thing is windows nodes vs linux nodes

MicroK8s will run in windows correctly, having the linux node installed and running by default (https://microk8s.io/docs/install-alternatives).

If the intention is to use Windows nodes (in order to run windows pods inside) (https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/) then you need to use Windows Server 2019. Microk8s doesn't run windows nodes, but does accept windows nodes to be joined to an already existing microK8s cluster.

So, answering your question, you can have microk8s installed in the win10 host directly (running the linux node it automatically starts with), and then connect to a VM (hyper-v, virtualbox, vmware player, etc), install there Windows server 2019 and the node side of https://discuss.kubernetes.io/t/add-a-windows-worker-node-to-microk8s/13782 and join it to the cluster.
Just make sure both machines can "see" each other.

In my particular case, I use a 5 VMs setup inside virtualbox with 3 linux (for testing high availability) and 2 Windows Server 2019, this way i can easily pause the whole cluster and get back to it when needed.

Hope it helps :)

[matt-psaltis]

Thank you very much for such a detailed explanation.

I'm definitely focusing on the Windows nodes side of things and given similarities between Windows 10 and Server 2019, I wondered if enough of the components were available to bend that requirement somewhat.

Thanks again!