K3s: Add buildah/podman for managing images

Created on 18 May 2019 · 21Comments · Source: k3s-io/k3s

Is your feature request related to a problem? Please describe.
I require the ability to re-tag and manage images (including pushing to a private registry) inside the k3s environment (typically at the master or single node).

Describe the solution you'd like
I would like to have either buildah or podman baked in, similarly to how crictl is currently made available.

Describe alternatives you've considered
Installing either separately.

Additional context
N/A

kinenhancement

Source

rcarmo

👍31

Most helpful comment

@sandys "Docker is pretty much not moving on cgroups v2" is totally false. The runc project is working towards getting cgroups v2 support working; this is a community project with many groups involved, including Docker, Suse and others. You can use Docker with crun which should probably work at this point for v2, although I have not tested this yet.

justincormack on 28 Oct 2019

❤2 👍2

All 21 comments

Any news?

hakuno on 9 Sep 2019

I'm very interested in this. Basically today k3s does not address the full flow of container development, namely building images. I think it may be companion project too. If I have some time I'll look into this. I would say that I wouldn't expect this soon because the effort is probably substantial. It would be significantly easier if podman, buildah was built on the same libraries as containerd, but they aren't.

ibuildthecloud on 11 Sep 2019

👍2

I'm interested on this because I would like to create k3s images which run e.g. full istio and knative inside a container. With dind (and kind) this is a bit cumbersome.

You may ask why would I want to run istio with k3s in container? For easy and reproducible learning setup.

Podman is great for many reasons, one is that you can pull during a creation of a image, e.g. in Dockerfile

RUN podman pull nginx

Same thing can't be done with dind because it requires dockerd to be running.

Ciantic on 21 Sep 2019

For me this I see entirely about image management in the cluster itself. When setting up a test/dev cluster, I find myself wanting to create containers on the more often than not, because I just don’t have hat many machines (or the right kind of machines) lying around.

rcarmo on 22 Sep 2019

@ibuildthecloud we are now very interested in this . Docker is pretty much not moving on cgroups v2 https://github.com/opencontainers/runc/issues/654

in order to force this issue, Fedora has made cgroups v2 as default and mandatory in the new upcoming Fedora 31 causing docker to fail to run. https://github.com/docker/for-linux/issues/665

Podman (and other docker equivalents) have supported cgroups v2 for years.

Being tied to docker is probably not the best idea at this point.

sandys on 22 Oct 2019

Rio already integrates image builder (buildkit) into k3s

AkihiroSuda on 22 Oct 2019

@sandys

Anyway Kubernetes doesn't support cgroup2 yet

AkihiroSuda on 22 Oct 2019

justincormack on 28 Oct 2019

❤2 👍2

@justincormack

Docker+crun+cgroup2 still doesn't work yet because shim doesn't support cgroup2.

PR: https://github.com/containerd/cgroups/pull/102

AkihiroSuda on 28 Oct 2019

Hi Justin
My comment was not intended as a blame, but simply highlighting status quo.
With the next Fedora drop, there is a high likelihood that we will be
locked out of docker.

And in the context of os vs docker...OS wins (because of overall security,
etc etc etc).

Sorry if the words sounded unfriendly

On Mon, 28 Oct, 2019, 17:15 Akihiro Suda, notifications@github.com wrote:

@justincormack https://github.com/justincormack

Docker+crun+cgroup2 still doesn't work yet because shim doesn't support
cgroup2.

PR: containerd/cgroups#102
https://github.com/containerd/cgroups/pull/102

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/k3s/issues/488?email_source=notifications&email_token=AAASYU4AUTBEYCPWJHMPASLQQ3GFRA5CNFSM4HN3EC3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECMS55Y#issuecomment-546909943,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYUZQT6VTZ5S45AO2IKDQQ3GFRANCNFSM4HN3EC3A
.

sandys on 28 Oct 2019

I’m just going to add a little note here that I’d be perfectly OK with something else but podman (in fact, in the months since I opened this issue, I’ve had a few issues with podman and would rather not rely on it).

rcarmo on 27 Dec 2019

@sandys

Anyway Kubernetes doesn't support cgroup2 yet

containerd and runc dont support cgroup2 yet, there are alternatives. I believe this issue is best for tracking containerd catching up.
https://github.com/containerd/cgroups/issues/104

leigh-j on 9 Apr 2020

containerd and runc dont support cgroup2 yet

Both supports cgroup2

AkihiroSuda on 9 Apr 2020

Well it seems to be a problem for me:

Apr 13 12:33:56 maciej.workbook k3s[182605]: time="2020-04-13T12:33:56.388735776+08:00" level=info msg="k3s is up and running"
Apr 13 12:33:56 maciej.workbook k3s[182605]: time="2020-04-13T12:33:56.388832649+08:00" level=warning msg="Failed to find cpuset cgroup, you may need to add \"cgroup_enable=cpuset\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Apr 13 12:33:56 maciej.workbook k3s[182605]: time="2020-04-13T12:33:56.388849130+08:00" level=error msg="Failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Apr 13 12:33:56 maciej.workbook k3s[182605]: time="2020-04-13T12:33:56.388865737+08:00" level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"

$ uname -a
Linux maciej.workbook 5.5.15-200.fc31.x86_64 #1 SMP Thu Apr 2 19:16:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ k3s --version
k3s version v1.17.4+k3s1 (3eee8ac3)

MaciejKucia on 13 Apr 2020

news solutions ?

johndiego on 22 Sep 2020

also interewsting in that

vitas on 26 Sep 2020

_Following thread._

Per PR https://github.com/k3s-io/k3s/pull/2584:

Support cgroup v2.

This PR does not add support for rootless cgroup yet. Rootless cgroup will be supported in a separate PR soon.

olinfc on 9 Dec 2020

FWIW we will probably be filling this gap by integrating https://github.com/rancher/k3c

brandond on 9 Dec 2020

👍1

Well, as long as there is a simple option for having a private registry, possibly even an in-cluster one that can work without Internet access - possibly also without TLS, since it is a major pain for IoT and industrial environments without connectivity - I'm good.

rcarmo on 9 Dec 2020

@dweomer assigning to you. Can you add or link to your design of k3c/k3b here?