K3s: Plans for ZFS support?

[fuzzykiller]

Deployment on ZFS is currently not possible, because OverlayFS does not work with ZFS:

overlayfs: filesystem on '/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/273/fs' not supported as upperdir

From the containerd logfile, it appears the ZFS snapshotter is not included:

time="2019-02-27T14:55:43.605823860+01:00" level=info msg="starting containerd" revision= version=1.2.3+unknown
time="2019-02-27T14:55:43.606278371+01:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2019-02-27T14:55:43.606418919+01:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2019-02-27T14:55:43.606671517+01:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2019-02-27T14:55:43.607001436+01:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2019-02-27T14:55:43.624241298+01:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
...

ZFS support would be awesome! Are there any plans to include that?

[ibuildthecloud]

We specifically removed the ZFS snapshotter. The reason being that we don't intend to include the ZFS user space as I believe that is not portable across kernel versions. So we can include the ZFS snapshotter and you would be required to first install the ZFS tools, which is common place already.

[fuzzykiller]

Thanks for the quick feedback!

The ZFS userspace tools are generally present when ZFS is used because otherwise ZFS cannot be used. So that’s really a non-issue as far as I can see. Except perhaps when running k3s in a Docker container on top of ZFS, but I’m not sure whether that would work anyway.

[fire]

Is there a timeline for including the ZFS snapshotter?

[jirkadanek]

ZFS is sadly quite painful with Docker in Docker and similar scenarios. It might be best to avoid the problem by creating a volume in your ZFS pool, formatting that volume to ext4, and having docker use "overlay2" on top of that, instead of "zfs".

zfs create -s -V 20GB zroot/docker
mkfs.ext4 /dev/zvol/zroot/docker
# add the mount to /etc/fstab
mount /dev/zvol/zroot/docker /var/lib/docker

The zfs create -s is for sparse volumes. Analogous to thin provisioning on LVM.

I just finished setting this up and it nicely solves my problems with k3s and also kind. I use these for testing and development, and there the volume should be just fine.

[terinjokes]

I was able to point k3s at the host's containerd (which had the zfs snapshotter and was configured to use it by default) and successfully run pods.

$ k3s server --container-runtime-endpoint /run/containerd/containerd.sock

$ sudo k3s kubectl -n kube-system get po -owide
NAME                                      READY   STATUS    RESTARTS   AGE     IP            NODE     NOMINATED NODE   READINESS GATES
local-path-provisioner-58fb86bdfd-tx9lt   1/1     Running   0          4m52s   10.88.0.137   rincon   <none>           <none>
coredns-57d8bbb86-7t6v6                   1/1     Running   0          4m52s   10.88.0.136   rincon   <none>           <none>

$ zfs list -r vault/storage/containerd
NAME                          USED  AVAIL     REFER  MOUNTPOINT
vault/storage/containerd     37.2M   466G       26K  /var/lib/containerd/io.containerd.snapshotter.v1.zfs
vault/storage/containerd/1    450K   466G      450K  legacy
vault/storage/containerd/10    25K   466G     20.0M  legacy
vault/storage/containerd/3     15K   466G      451K  legacy
vault/storage/containerd/4     15K   466G      451K  legacy
vault/storage/containerd/5    238K   466G      238K  legacy
vault/storage/containerd/6   3.67M   466G     3.67M  legacy
vault/storage/containerd/7   19.8M   466G     20.0M  legacy
vault/storage/containerd/8   13.0M   466G     16.6M  legacy
vault/storage/containerd/9     35K   466G     16.6M  legacy

[haohaolee]

@terinjokes Hi, could you please share more details about containerd with zfs support. How can I install containerd with zfs support?

[jirkadanek]

Another possible workaround, again it involves creating ext4 filesystem, but this time using Docker itself through Docker loopback volume plugin, not by asking zfs to do it. Described in rancher/k3d docs https://github.com/rancher/k3d/blob/dc4c29361fd433d3592b62d3d8d23256a5ed5728/docs/examples.md#running-on-filesystems-k3s-doesnt-like-btrfs-tmpfs-

[stevefan1999-personal]

Well you can apparently leverage the host's docker for k3s, which can spawn containers on a ZFS-based graph.
You just need to add --docker argument to k3s server/agent. Make sure that your docker info is using ZFS tho.

ExecStart=/usr/local/bin/k3s \
    server --docker\

This however completely defeats the purpose of packaging CRI in k3s.

[stevefan1999-personal]

We specifically removed the ZFS snapshotter. The reason being that we don't intend to include the ZFS user space as I believe that is not portable across kernel versions. So we can include the ZFS snapshotter and you would be required to first install the ZFS tools, which is common place already.

This is not necessary the case now for Proxmox-based distribution with version >= 5, and also now in Ubuntu 20.04 (albeit deemed "experimental"), both of which cherry-picks and integrates OpenZFS Linux module codebase into their kernel recipe.

OpenZFS ABI is very stable but is denied entering kernel mainline due to some stupid license politics problem. And even if it is unstable, you can always use DKMS to hopefully keep it up with the new kernel.

Despite this, I think k3s team still won't consider adding ZFS support anyway, but what about re-adding AUFS support? AUFS works generally fine atop of ZFS despite there are some reported edges cases related to file mounts not in sync, and AUFS is more general than ZFS and easier to merge in k3s I think.

EDIT:

aaaaaaaaaaaand btw the current setback is to create a ext4 volume as containerd backend like @jirkadanek did, but on a different location

# echo "[+] making sure everything in containerd is empty to let it mount"
# rm -rf /var/lib/rancher/k3s/agent/containerd/*
# echo "[+] create the k3s hierarchy"
# zfs create rpool/k3s
# zfs create -s -V 128GB rpool/k3s/containerd
# echo "[+] wait for the newly created volume to format"
# mkfs.ext4 /dev/rpool/k3s/containerd
# echo "[+] adding the mount to fstab"
# echo "/dev/rpool/k3s/containerd /var/lib/rancher/k3s/agent/containerd ext4 auto 0 0" > /etc/fstab
# echo "[+] manually mounting the volume to the specific mount point"
# mount /dev/rpool/k3s/containerd /var/lib/rancher/k3s/agent/containerd
# echo "[!] if you saw this message without any error it means you're good to go"

Keep in mind that this is still not perfect -- so you basically recreated snapshot on ZFS, and so the atomic COW goodies is probably gone, you have no fine control over the data size whatsoever, I think in the future maybe we could get to install k3s plugins to accommodate situations like these.

ZFS is a very popular filesystem among Proxmox users after all, and it's a shame such a beautiful piece of work is being insulted like this.