K3os: K3OS image for LXC

[rvdmei]

I’ve been using k3s on LXC for local experiments with Kubernetes (cluster in a box). Having an LXC image with k3os would make setting up a cluster in a box a lot easier / faster.

[ibuildthecloud]

I'm not opposed to this but not sure if there is specific advantage of running k3os in LXC over just running k3s on say an Ubuntu image in LXC. For this to work in LXC I assume you need to set the "privileged" options in LXC right?

[rssfed23]

There's also a usecase here for proxmox users - enabling a k3s lxc privileged image would let people build clusters within a couple of clicks

[rvdmei]

I think k3os is an ideal solution for cluster-in-a-box (See this kubcon video https://youtu.be/kQslklE5dKs). With an LXC image, setup and maintenance would be simplified.

[stevefan1999-personal]

This won't work due to the lack of iptables module in LXC userspace that k3s/k8s will ultimately fail at network configuration state, and you will have to wait forever for health to bring up, which will die and wait and again.

I tested it manually with futile in the end.

[laxmanpradhan]

This won't work due to the lack of iptables module in LXC userspace that k3s/k8s will ultimately fail at network configuration state, and you will have to wait forever for health to bring up, which will die and wait and again.

I tested it manually with futile in the end.

I believe this issue has been fixed in the kernel 5.3 update: https://github.com/lxc/lxd/issues/5193#issuecomment-431872713A

My proxmox 6.1-7 is showing: kernel 5.3.13-1-pve

I'll try to test this using alpine LXC containers and the k3s package for alpine.

[ShahBinoy]

This won't work due to the lack of iptables module in LXC userspace that k3s/k8s will ultimately fail at network configuration state, and you will have to wait forever for health to bring up, which will die and wait and again.
I tested it manually with futile in the end.

I believe this issue has been fixed in the kernel 5.3 update: lxc/lxd#5193 (comment)

My proxmox 6.1-7 is showing: kernel 5.3.13-1-pve

I'll try to test this using alpine LXC containers and the k3s package for alpine.
@laxmanpradhan
Were you able to test this out ?

[laxmanpradhan]

This won't work due to the lack of iptables module in LXC userspace that k3s/k8s will ultimately fail at network configuration state, and you will have to wait forever for health to bring up, which will die and wait and again.
I tested it manually with futile in the end.

I believe this issue has been fixed in the kernel 5.3 update: lxc/lxd#5193 (comment)
My proxmox 6.1-7 is showing: kernel 5.3.13-1-pve
I'll try to test this using alpine LXC containers and the k3s package for alpine.
@laxmanpradhan
Were you able to test this out ?

I tested it with proxmox 6.1-7 and an ubuntu 19.10 and alpine 3.10 LXC container and it did not work. Proxmox does not have br_netfilter module installed by default, so it gives an error about not being able to load that module. May be possible to add it to /etc/modules-load.d/modules.conf on the proxmox host but I don't want to try it on my only server.

[admiralakber]

This won't work due to the lack of iptables module in LXC userspace that k3s/k8s will ultimately fail at network configuration state, and you will have to wait forever for health to bring up, which will die and wait and again.
I tested it manually with futile in the end.

I believe this issue has been fixed in the kernel 5.3 update: lxc/lxd#5193 (comment)
My proxmox 6.1-7 is showing: kernel 5.3.13-1-pve
I'll try to test this using alpine LXC containers and the k3s package for alpine.
@laxmanpradhan
Were you able to test this out ?

I tested it with proxmox 6.1-7 and an ubuntu 19.10 and alpine 3.10 LXC container and it did not work. Proxmox does not have br_netfilter module installed by default, so it gives an error about not being able to load that module. May be possible to add it to /etc/modules-load.d/modules.conf on the proxmox host but I don't want to try it on my only server.

Hi everyone, I think k3os as a template would be great.

@laxmanpradhan - I tried this on my server. No luck.
According to a post on the Proxmox Forums:
https://forum.proxmox.com/threads/docker-support-in-proxmox.27474/page-6#post-295237
apparently br_netfilter is not needed in PVE Kernels as: CONFIG_BRIDGE_NETFILTER=y (see below)

I have posted this information and a link to this issue on the Proxmox Forums here:
https://forum.proxmox.com/threads/missing-br_netfilter-module.53791/

root@pve:~# modinfo br_netfilter
modinfo: ERROR: Module br_netfilter not found.

root@pve:~# lsmod | grep br_netfilter
## returns nothing

root@pve:~# find /lib/modules/ -type f -iname '*br_netfilter.ko'
## returns nothing

root@pve:~# cat /etc/modules-load.d/k3s-lxc.conf
br_netfilter

root@pve:~# modprobe br_netfilter
## returns nothing

root@pve:~# grep 'BRIDGE_NETFILTER' /boot/config-$(uname -r)
CONFIG_BRIDGE_NETFILTER=y

[haohaolee]

FYI, I have a workaround to make k3s run in a LXC container.
First make sure docker can be running in a LXC container (by using a privileged container), there are many discussions about this. Seems docker would not bother with br_netfilter stuff.
Then, install k3s using --docker flag

This is far from perfect, but I can use k3s in a LXC container now. I am not sure if this is helpful for creating a template for k3os

[atif1996]

I’m surprised you guys have had trouble. We have been running k3s in LXC in production for the last two years.

[CuBiC3D]

I’m surprised you guys have had trouble. We have been running k3s in LXC in production for the last two years.

This is the k3os repo.