Runtime: Implement containerd shim v2 API for Kata Containers

[gnawux]

In the arch committee meeting last week, Michael Crosby introduced the containerd shim API v2. After the meeting, read related documents, patches, and discussed with @bergwolf @lifupan and related folks. And I think we could begin to implement the kata shim for containerd v2 API now.

Current kata & containerd working pattern

Right now, containerd and runtime work in the following pattern:

Whenever containerd tries to create a container,

• It creates a containerd-shim
• The containerd-shim call runtime cli -- the cli is specified by the runtime=kata flag, and the cli should be compatible with runc
• The containerd-shim served at the address specified by containerd, and feedback events to containerd’s listening address

Though the containerd-shim designed to be an replaceable component

• There still has to be one containerd shim per container (or process?)

Benefits from shim v2 API

With the proposed shim v2 API, the most significant change is

• The shim will write the serving address back to containerd through stdout, instead of specified by containerd

As a result

• We may replace the containerd-shim with a customized shim, and don’t need to implement a runc compatible CLI for containerd-shim.
• We may use a single shim for a couple of containers, such as one shim per pod

Then the architecture may change to the follows:

• One shim for containerd per pod
• No proxies and kata-shim processes any more

And in v2 API, stats function is moved to shim, which makes the shim more self-contained.

Where we start

• This routing pattern (built-in proxy + built-in shim) has already been implemented in hyperhq/hyperd#727
• The OCI Spec processing part exists in kata runtime
• And the other stuff could reference the current kata-shim and v2 shim PR (containerd/containerd#2434).

[WeiZhang555]

This is a good proposal, and simplifies the architecture a lot!

One question: I think we need to keep the compatibility with older containerd-shim, we can't only support containerd-shim V2 because it's so new.

[bergwolf]

@WeiZhang555 I agree we should keep the runc compatible cli as long as possible. For this new shim, IMO it should have its own repo so that in future we can push it to containerd to be a first class citizen there.

[sboeuf]

As said during the Arch committee meeting, I would like to see the Kata implementation of the containerd-shim v2 as a separate directory (different repo might come further down the road, but right now, let's keep this simple). The same way we have the cli directory for our OCI implementation (matching runc as much as possible), we would have a new containerd-shim-v2 directory providing the Kata implementation of this shim v2. Most of the implementation will follow what frakti is actually leveraging (stateless Kata API), to be more efficient.
@gnawux let us know who will work on that, and how we can help with this (if there are some areas where the work can be split).

[crosbymichael]

Looks good!

[WeiZhang555]

And as @crosbymichael described in the Arch Committee meeting, containerd-shim-v2 isn't aware of POD concept, so the containerd-shim should be one per container, so I think the final arch should be:

Is this right? @gnawux

[gnawux]

@WeiZhang555 we investigated the APIs and code in the PR and has the following conclusion -- though containerd will request shim service for every container, it does not expect there is a new shim socket or process for the container. As a result, we think it could be implemented as one shim per pod. Let's just try to prove it with a prototype.

[bergwolf]

@WeiZhang555 Per discussions in containerd/containerd/issues/2426, containerd-shim-v2 start can return the unix socket address of a previously started shim. That's how one shim per pod is supposed to be implemented.

[WeiZhang555]

Oh, cool, this sounds a little hacky but workable.

[crosbymichael]

Yes, that is correct. We updated the v2 API to introduce a start command for the shims. This way the shims can configure and launch new shims how they need to. It also allows this start command to inspect the spec, look at annotations for pods, and return an existing shim instead of creating a new one. If one does not already exist, it can start one.

[crosbymichael]

I pushed an README this morning for shim authoring. Please take a look and provide any feedback if you have questions or if something does not make sense.

https://github.com/containerd/containerd/pull/2434/files#diff-9bb937a499e2e6fdc4536481d6315c50

[gnawux]

thank you @crosbymichael. One more question -- is it essential for a v2shim commandline to implement the "no action" behavior

    default:
        client := NewShimClient(ctx, service, signals)
        return client.Serve()

or it's ok to support start and delete only?

[crosbymichael]

If you want to split it into multiple binaries then that is fine, all we care about is that start returns a socket to the API and delete lets us call for a cleanup if a shim dies and we can no longer communicate to it.

[resouer]

[Update] A temporary branch has been created in develop repo here.

Will send out PRs to upstream when core apis are ready so you can set up basic tests.

[grahamwhaley]

Nice @resouer . Just for ref, link to branch itself is here I think? (link above goes to closed PRs in that repo afaict).

[resouer]

@grahamwhaley Oh, yes. The link I posted seems to be the merged PRs to that branch :/

[Ace-Tang]

Looking forward to the implement !

[gnawux]

@Ace-Tang PR #572 has already been there under review. Comments are welcome