Microk8s: Error out of disk space when install in a LXC container

[raguilera82]

Hi,

I have an error when I am trying to install microk8s in a LXC container with nesting and privileged security to true. I follow these steps:

1. After access the container, I install it: sudo snap install microk8s --edge --classic
2. Enable dns and dashboard addons: microk8s.enable dns dashboard
3. List all pods: microk8s.kubectl get pods --all-namespaces
4. Access info heapster pod: microk8s.kubectl describe pod heapster-v1.5.2-84f5c8795f-c8hld --namespace kube-system

At that moment I see this info:

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  21m (x6 over 21m)  default-scheduler  0/1 nodes are available: 1 node(s) were not ready.
  Warning  FailedScheduling  21m                default-scheduler  0/1 nodes are available: 1 node(s) were not ready, 1 node(s) were out of disk space.

I have checked my ZFS storage and I have not a problem with my disk space.

Can anyone reproduce it with LXC container? Any ideas?

Thank you very much

[ktsakalozos]

Hi @raguilera82 ,

What is your lxc profile look like?

I had partial success in running microk8s with this one:

name: microk8s-profile
config:
  boot.autostart: "true"
  linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  aadisable:
    path: /sys/module/nf_conntrack/parameters/hashsize
    source: /dev/null
    type: disk
  aadisable1:
    path: /sys/module/apparmor/parameters/enabled
    source: /dev/null
    type: disk

[ktsakalozos]

@raguilera82 , seems docker inside lxc containers does not play well with ZFS https://github.com/lxc/lxd/issues/2305 could you use dir instead?

[raguilera82]

Hi @ktsakalozos

Thanks for your answers. My experience with LXC containers (version 3.2) is that they work properly with Docker and ZFS (this is the recommend storage) if you enabled nesting and privileged properties of security. For example, I have a LXC container with an instance of GitLab and I can work with Docker and docker-compose without problems into the container. The issue that you have mentioned is quite old.

Now, I can not change the system storage, maybe for that reason I will get back to Minikube until this issue will be fixed.

Thank you very much for your time.

[ktsakalozos]

Thank you for giving microk8s a try

[ktsakalozos]

Hi @raguilera82 ,

After some experimentation with LXC and zfs I got microk8s working. Here is what I had to do:

1. Create the following profile:

$ cat microk8s.profile 
name: microk8s
config:
  boot.autostart: "true"
  linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,nf_conntrack_ipv4,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw
    lxc.cgroup.devices.allow=a
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  aadisable:
    path: /sys/module/nf_conntrack/parameters/hashsize
    source: /sys/module/nf_conntrack/parameters/hashsize
    type: disk
  aadisable1:
    path: /sys/module/apparmor/parameters/enabled
    source: /dev/null
    type: disk
  aadisable2:
    path: /dev/zfs
    source: /dev/zfs
    type: disk
$ lxc profile copy default microk8s
$ cat microk8s.profile | lxc profile edit microk8s

1. Launch a container:

$ lxc launch -p default -p microk8s ubuntu:18.04 

1. Install zfsutils-linux in the container. This package should be part of microk8s so I also opened a new issue.

# apt install zfsutils-linux

1. And finally install microk8s

# snap install microk8s --edge --classic

Most of the work is on the lxc profile so there is not really much to do for microk8s. Here is an ascii cast of the above steps: https://asciinema.org/a/198964

Thanks

[raguilera82]

Thank you @ktsakalozos

Your steps work perfectly! Also, I have tried with ubuntu:16.04 with your microk8s profile and all works like a charm.

Thank you very much for your time!

[kim0]

@ktsakalozos Thanks for the info above! I have a question, I see we're removing a lot of the security constraints from lxc .. Can you comment on how contained is that lxc container now? Is it still a little hard to break out of a lxc container with that configuration, or does it basically have full host control without much effort? Basically can I let fairly hostile guests in that lxc with some level of guarantee it's not too easy to break out

Thanks!

[ktsakalozos]

Hi @kim0 ,

I am not entirely sure what you are asking me. I would say it is a little hard to reach the host but not hard enough.

MicroK8s, can get out of the lxc confinement and can access the host machine and this is the purpose of the profile we apply. MicroK8s is expected to run on the host and you are expected to trust it. What you might not trust is the user workload that runs inside Kubernetes and that is constrained by docker containers. You should be aware that MicroK8s is not supposed to be deployed in production or in untrusted environments. Its purpose is mainly to assist in application development locally. If security is of concern you should look into production grade Kubernetes solutions.

Having said that here are a few ways to harden MicroK8s.

• Refine the lxc profile
• Use MicroK8s inside a VM
• Set AppArmor and firewall rules that apply to your usecase
• Test and contribute to this PR https://github.com/ubuntu/microk8s/pull/88

Thanks