Kind: support multiple container engines on the host

[BenTheElder]

#151 should bring initial support for podman via @rhatdan

~This should mostly be a drop in replacement.~ (it's not... podman has quirks)

We should further clean up the code base a bit to help with this sort of thing. We've mostly but not quite isolated the container runtime functionality.

/assign
/assign @rhatdan

/kind feature
/kind cleanup
/priority important-longterm

related: #153 is the same thing, but within the nodes

[BenTheElder]

There's now a container package with docker as a subset, and we're working on plumbing through optional arguments to the cluster better, I expect to get limited podman support based on these pretty soon.

[BenTheElder]

We're also targeting CRI functionality explicitly: https://kind.sigs.k8s.io/docs/design/principles/#target-cri-functionality

[BenTheElder]

Currently working on moving more of the flags (ports) behind abstractions borrowed from CRI.
I'd like to isolate all knowledge of docker / podman style flags to pkg/container/... by the next milestone, at which point we should be able to do podman support properly with it's own implementation of all functionality.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tao12345666333]

/remove-lifecycle stale

[tinyzimmer]

This is pretty old, but in case anyone is still paying attention - I'm pretty close to having this working where basically the global --podman switch uses podman instead.

I had to do something along the lines of what you are describing first, @BenTheElder, where I made a generic Container interface and different backends (e.g. docker,podman) just need to implement the various methods.

package runtime

// .. import pkg/container/backends ...

type ContainerRuntime string

const Podman ContainerRuntime = "podman"
const Docker ContainerRuntime = "docker"

var currentRuntime ContainerRuntime

func SetContainerRuntime(rt ContainerRuntime) {
    currentRuntime = rt
}

func GetContainerRuntime() container.Container {
    switch currentRuntime {
    case Docker:
        return docker.New()
    case Podman:
        return podman.New()
    default:
        panic("invalid container runtime")
    }
}

Then replaced all the places that called the docker package directly with basically a runtime.GetContainerRuntime()

[BenTheElder]

Yeah, sorry this has been drawn out. We're definitely doing this, things have settled a bit and now that I've landed some new debugging infrastructure and other major cleanups I'll be reviving this ...

That said: If you're just replacing docker with podman in all the commands that should be possible today without any changes to the binary:

TMP_DIR=$(mktemp -d)
ln -s "$(which podman)" "${TMP_DIR}/docker"
export PATH="${TMP_DIR}:${PATH}"
kind create cluster

We'd like to do something a little more comprehensive to make things more flexible internally and go beyond even just docker and podman eventually (eg kata or ignite might be quite handy for some use cases) ...

[tinyzimmer]

I tried that at first and I couldn't get it to work actually, linux and wsl2 (which is its own ball game entirely). And I might have spoken too soon on my previous comment, as well. Podman...has its quirks still.

So far it was just simple things like format templates being a bit different, but then I ran into an issue with the node failing to start because it couldn't remount /sys - no matter what flags I tried. Running with sudo works, but hopefully I can figure out the real cause of that.

Now for some reason node.WriteFile() when used with podman is not returning an error, but also not writing a file...

This might be an adventure still. But in the spirit of doing something comprehensive, I do at least have it closer in that direction and it doesn't break any of the docker capabilities. It's kind a dud of a branch though if I can't even get a cluster built with podman 😛.

EDIT: I think the next logical step in trying to accomplish this would be to swap the CLI calls with native API calls. github.com/docker/client, github.com/containers/libpod, etc...

There is a regression in podman > 1.5.0 with passing stdin to the container during an exec. I'm getting further with 1.4.4 though.

[BenTheElder]

I have a pretty drastic refactor to put node creation / deletion / interaction behind interfaces ongoing in this branch: https://github.com/BenTheElder/kind/tree/nodes

[BenTheElder]

PR for that branch https://github.com/kubernetes-sigs/kind/pull/879

[BenTheElder]

879 is in, we're _most_ of the way to having this abstracted out for nodes, there's one or two code paths that still depend on specifically docker. I'm working on a follow up to eliminate the rest of those.

[BenTheElder]

https://github.com/kubernetes-sigs/kind/pull/1023 is the last change to remove docker from the critical path for bringing up a cluster and move it behind the provider interface.

I intend to do further cleanup to the APIs around this and experiment with podman etc. in the v0.7 timeframe (AKA O(soon)). We're almost done with the fixes for 0.6

[BenTheElder]

1023 landed, #1025 finishes removing docker specific things from the APIs, leaving room to start experimenting with Provider implementations.

I've also fired up a throwaway GCE VM with podman for a while now, doing some experiments with that to get a better understanding of what supporting podman properly would entail now.

[rhatdan]

Awesome.

[christianh814]

Since Fedora 31 basically broke docker for me, I'm just adding this comment so I can follow this. I've been using podman for everything EXCEPT KinD. So this will be great to only use one tool.

@rhatdan - Is there a corresponding being tracked at the podman repo?

As soon as there's things to test, I'll be ready! :)

[BenTheElder]

Since Fedora 31 basically broke docker for me, I'm just adding this comment so I can follow this. I've been using podman for everything EXCEPT KinD. So this will be great to only use one tool.

So, Fedora 31 broke _kubernetes_. Podman will not fix this. You need to switch to cgroups v1, which will unbreak docker and Kubernetes.

[BenTheElder]

In theory it should be possible to test podman compatibility with kind by doing:

TMP_DIR=$(mktemp -d)
ln -s "$(which podman)" "${TMP_DIR}/docker"
export PATH="${TMP_DIR}:${PATH}"
kind create cluster

With zero kind changes.

But in practice you'll likely find that podman is still not actually 100% compatible to the docker CLI, so we'll need to start implementing a custom backend in kind to handle these.

I have a GCE vm with podman starting on this but it is NOT the priority of this project. The focus is testing Kubernetes for the Kubernetes project, which we are accomplishing with docker for the moment. https://kind.sigs.k8s.io/docs/contributing/project-scope/

[christianh814]

Since Fedora 31 basically broke docker for me, I'm just adding this comment so I can follow this. I've been using podman for everything EXCEPT KinD. So this will be great to only use one tool.

So, Fedora 31 broke _kubernetes_. Podman will not fix this. You need to switch to cgroups v1, which will unbreak docker and Kubernetes.

ACK. I will be passing the kernel boot options for now.

[tinyzimmer]

I am like 90% done with a podman provider with @BenTheElder 's refactor.

The aforementioned rootless issue still applies. I can't get the kind image to boot properly unless podman (and therefore kind) is executed as root. This also causes the kubeconfig to not get merged properly if you are wanting a non-root user to use kubectl. If someone is wanting to take a swing at this, make sure you first allow your user a crap-load more uids than one might normally would.

E.g. my /etc/sub{u,g}id is

devbox:10000:344930

just to get the container created in rootless podman.

The issue I'm trying to figure out right now is with cleanup. Specifically, the pods in kube-system are somehow persisting across creates and deletes of clusters and I'm not sure why at the moment. This could just be something I'm not grasping about podman, but figured I'd drop in here if any one knows what might cause something like that.

[BenTheElder]

/assign @amwat
We just ~paired on this for a bit, we have something running but the above issues are still true and need verified workarounds :-)

Expect PRs related to this issue soon.

The issue I'm trying to figure out right now is with cleanup. Specifically, the pods in kube-system are somehow persisting across creates and deletes of clusters and I'm not sure why at the moment. This could just be something I'm not grasping about podman, but figured I'd drop in here if any one knows what might cause something like that.

That's because podman until October last year didn't support anonymous volumes properly, instead it bind mounted them to the same path on the host (!)

https://github.com/containers/libpod/pull/4287

[amwat]

More updates: We'll need to wait for podman 1.8 to get
https://github.com/containers/libpod/pull/4818 that fixes a regression in podman exec.
we use exec heavily, e.g. https://github.com/kubernetes-sigs/kind/blob/4c7854c1347860ef04a7a8caef625ac4491afa4b/pkg/cluster/nodeutils/util.go#L76

[BenTheElder]

can confirm that it's workable with 1.7.1-dev though, needs some more work in kind but feasible once https://github.com/containers/libpod/pull/4818 lands.

[rhatdan]

It has landed and hopefully will be in podman 1.8.0.

[rhatdan]

BTW Now that Podman is beginning to support the Docker API, it would be interesting to see if KIND could use it.

[BenTheElder]

https://github.com/kubernetes-sigs/kind/pull/1302 -- provisional support is in, it requires a dev build (1.7.0-dev+) until 1.8.0 is out, then we'll probably start requiring 1.8.0. It also requires rootful for now.

[BenTheElder]

In v0.8.0 we will be selecting podman or docker backends automatically. The support is a bit rough (needs rootful, which makes some things clunky) but mostly functional.

There will be more details in the release announcement, probably relatively soon. I'd like to fix up a few smaller things in 0.8

[mathieu-benoit]

Hi @BenTheElder, just curious, any updates about the 0.8.0 release and/or the support of CRI containers runtime (podman)?

[BenTheElder]

There's an issue tracking the 0.8.0 release #1475

Note that the podman support is experimental and has many limitations, I don't recommend it yet and we're not committed to keeping it for sure, certain desirable features seem to be infeasible (like #148)

[mathieu-benoit]

Good to know, thanks for the clarification.

[BenTheElder]

Just wanted to update that we've all discussed this and are going forward with committing to the podman backend for now.

It's still under tested and secondary, notably host-reboot functionality will likely be lacking for the immediate future, but we have more core contributor commitment to this.

You can see evidence of this in https://github.com/kubernetes-sigs/kind/issues/1611 https://github.com/kubernetes-sigs/kind/issues/1581 https://github.com/kubernetes-sigs/kind/issues/1506 https://github.com/kubernetes-sigs/kind/issues/1567 https://github.com/kubernetes-sigs/kind/issues/1401#issuecomment-601457133 ...

[afbjorklund]

I would like to cooperate (minikube with kind), regarding the podman support.

We have added a "podman" driver to minikube next to the docker driver,
and a (varlink) "podman-env" next to the minikube docker-env command.

Since the "kicbase" we are using is based on kindbase, trying to upstream all

[BenTheElder]

xref: https://github.com/kubernetes-sigs/kind/pull/1639, thanks for the PR! cc @aojea @amwat ^^