Podman: [Discussion] podman and cri-o shared codebase

[saschagrunert]

Hey :wave:, I heard of the plan of integrating libpod into cri-o and want to support that. So I started digging through the libpods' sources and am now wondering where to start and especially how. Both projects seem to have a common foundation of functionality but have diverged over time.

One example would be comparing both files cri-o/pkg/storage/runtime.go and libpod/libpod/storage.go. In cri-o we already have this nice RuntimeServer interface, which is great for abstraction and testing, so it would be useful for libpod, too. On the other side the RuntimeContainerMetadata as well as the implementation for RuntimeServer already contains information about Pods, which happens in libpod on a higher level. To get the best (and working) out of both we would have to touch both projects nearly in parallel which makes it quite complicated in my point of view.

How to achieve this without breaking anything? One way would be leave libpod nearly untouched and glue it together on the cri-o side, but this seems to have negative implications for cri-o especially from a performance perspective. Another way could be to plan the migration on top of both projects and put the code used by both (like containers/{image,storage}) in a dedicated repository. I guess this was already the intention of libpod right?

I'm wondering If you already have a big plan on mind or written down somewhere. :)

Edit: meant runtime.go and not image.go. :man_facepalming:

[mheon]

For your first example - it's curious that you mention storage.go, because we originally sourced that from CRI-O (with a few changes). I also wouldn't compare it to image.go - our image code lives in libpod/libpod/image, so I think it's most similar to some of what we're doing with image.Image.

The current plan is to mostly supercede CRI-O's lib package, replacing it with libpod. The complexity here is going to be achieving feature parity between the two and maintaining the performance of CRI-O. Some of the new features CRI-O is merging (the runtime server work you mentioned, for example) have no parallel in Libpod yet. I don't think it would be that difficult to adapt (I think the biggest change is Podman/Libpod's multiprocess architecture, which means you can't really rely on in-memory state - the runtime server would need to store things in the DB for persistence).

[saschagrunert]

For your first example - it's curious that you mention storage.go, because we originally sourced that from CRI-O (with a few changes). I also wouldn't compare it to image.go - our image code lives in libpod/libpod/image, so I think it's most similar to some of what we're doing with image.Image.

My fault, I meant runtime.go. Fixed it within my initial comment.

[rhatdan]

@saschagrunert I am hoping to kick off the effort for the merge, as soon as Podman remote is completed.
First step is to get full image support from libpod into CRI-O.
But we would love all the help we can get.

[rhatdan]

Getting closer.

[saschagrunert]

Do you think it would be a good approach to separate the shared codebase between podman and CRI-O into a dedicated repository (library only)?

[haircommander]

@saschagrunert yeah I think separating libpod and other backend libraries into its own repo with its own visioning system is going to be important

[saschagrunert]

@saschagrunert yeah I think separating libpod and other backend libraries into its own repo with its own visioning system is going to be important

Cool, so we could start with the low level internals of libpod and CRI-O, like imageService and runtimeService, which directly interact with containers/{storage,image}. So we could built-up a clean middle-layer API. WDYT?

[rhatdan]

@saschagrunert @mheon @vrothberg Any movement on this issue?

[mheon]

We really need to get a meeting scheduled with all involved parties and get some discussion going on the way to go forward, but IMO the priority on this is rising steadily - we're spending too much effort developing them as two separate projects when they do fundamentally the same thing.

[saschagrunert]

We really need to get a meeting scheduled with all involved parties and get some discussion going on the way to go forward, but IMO the priority on this is rising steadily - we're spending too much effort developing them as two separate projects when they do fundamentally the same thing.

Yeah I agree. My proposal would be to use a dedicated project board on GitHub to increase the transparency about what is going on behind the scenes.

[vrothberg]

@mrunalp was thinking about scheduling a meeting in the near future.

[vrothberg]

First step is to get full image support from libpod into CRI-O.

^ this is still valid btw. The critical part is the runtime part while the image part is relatively straight forward. Ideally, we use that effort to consolidate the code of CRI-O, Podman and Buildah. There is a lot of redundancy among the projects w.r.t pushing and pulling that I would love to consolidate into one API that can be shared. Given the dependency chain, we _could_ add a high-level API to c/image (i.e., a wrapper around c/image/copy with the all the nasty special casing).

But as mentioned many times before: we need to know the exact requirements of CRI-O (see https://docs.google.com/document/d/1GKub_NvdGCo7veR8XXf5w3PaEpIwzU6TnTMKNDcErEA/edit?usp=sharing). The scope regarding images is limited, but somebody needs to start collecting what we need, then we can discuss how we can do it. What I see is that everybody wants it but nothing is happening.

[saschagrunert]

But as mentioned many times before: we need to know the exact requirements of CRI-O (see https://docs.google.com/document/d/1GKub_NvdGCo7veR8XXf5w3PaEpIwzU6TnTMKNDcErEA/edit?usp=sharing). The scope regarding images is limited, but somebody needs to start collecting what we need, then we can discuss how we can do it.

To be honest I don't think that a gdoc is the right way we can collaborate on that topic in an agile way. I agree that gathering the top requirements and bundling them together in some EPICs is a necessary step. What do you mean with "exact requirements"? How exact should they be?

I still believe that the Kubernetes Container Runtime Interface is a nice guidance in writing user stories around the topic, but you may have other plans... :man_shrugging:

What I see is that everybody wants it but nothing is happening.

I don't think that this is right as I know that multiple people spend hours in thinking hardly about this topic. :)

[vrothberg]

But as mentioned many times before: we need to know the exact requirements of CRI-O (see https://docs.google.com/document/d/1GKub_NvdGCo7veR8XXf5w3PaEpIwzU6TnTMKNDcErEA/edit?usp=sharing). The scope regarding images is limited, but somebody needs to start collecting what we need, then we can discuss how we can do it.

To be honest I don't think that a gdoc is the right way we can collaborate on that topic in an agile way.

Why not share this feedback in the first place? We all have been chatting about this doc since April and it was meant to discuss specific issues, and so far we did not fill out the tables. I have no tool preference as long as we're not turning in circles but I am convinced the tool shouldn't block the work. In the end, the doc was not meant for planning specific epics but for figuring out if libpod's APIs meet CRI-O's needs and extract work items from that.

I agree that gathering the top requirements and bundling them together in some EPICs is a necessary step. What do you mean with "exact requirements"? How exact should they be?

See https://github.com/containers/libpod/pull/2953 and the comments (https://github.com/containers/libpod/pull/2953#issuecomment-483677028 in particular, excluding the testing issue). The tables in the doc were a good start to figure out what's missing.

What I see is that everybody wants it but nothing is happening.

I don't think that this is right as I know that multiple people spend hours in thinking hardly about this topic. :)

And I _really_ appreciate it :pray: but the discussion around that topic pre-dates this issue and is ongoing for over a year and is turning in circles.

[saschagrunert]

See #2953 and the comments (#2953 (comment) in particular, excluding the testing issue). The tables in the doc were a good start to figure out what's missing.

Then let's stick to that and let's fill it with life. :+1:

[saschagrunert]

I updated the document regarding the migration analysis. I also added a coarse grained _Migration Plan_ to outline the necessary steps in their order.

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[vrothberg]

We have a team meeting next week where we will discuss the issue. @saschagrunert, hopefully we come up with a plan to move things forward where we can collaborate across the projects, teams and companies :) We'll reach out.

[saschagrunert]

We have a team meeting next week where we will discuss the issue. @saschagrunert, hopefully we come up with a plan to move things forward where we can collaborate across the projects, teams and companies :) We'll reach out.

Sounds good, thanks for the update @vrothberg :blush:

[rhatdan]

Perhaps we could setup a Blue Jeans for @saschagrunert to join.

[vrothberg]

Any objections against closing this issue? I think we have a consensus on how to move forward since the meeting in November.

We are about to start consolidation the image-related code; initially we planned for a high-level API in c/image but changed plans to create a new libpod/pkg/image (or similar) to consolidate as much high-level code in libpod as possible.
Once the API v2 work is done, libpod and podman can be put into separate projects which allows for a proper release management of the two.
Once libpod and podman are separated, we can open a branch over at CRI-O to migrate on libpod.

[saschagrunert]

We are about to start consolidation the image-related code; initially we planned for a high-level API in c/image but changed plans to create a new libpod/pkg/image (or similar) to consolidate as much high-level code in libpod as possible.
Once the API v2 work is done, libpod and podman can be put into separate projects which allows for a proper release management of the two.
Once libpod and podman are separated, we can open a branch over at CRI-O to migrate on libpod.

Is this anywhere written down? If not then it's nearly impossible for me to contribute to this plan.The only thing I can see so far is in https://github.com/cri-o/cri-o/projects/1.

Feel free to close if there needs nothing to be done here.

[vrothberg]

Is this anywhere written down? If not then it's nearly impossible for me to contribute to this plan.The only thing I can see so far is in https://github.com/cri-o/cri-o/projects/1.

The project seems to cover all relevant items. @mtrmac, once you start with the image work, would you move the card in https://github.com/cri-o/cri-o/projects/1 to "in progress"?

Feel free to close if there needs nothing to be done here.

:+1:

[saschagrunert]

Do we want to refine the tasks written down in https://github.com/cri-o/cri-o/projects/1 to be able to split them across multiple people?

[vrothberg]

Do we want to refine the tasks written down in https://github.com/cri-o/cri-o/projects/1 to be able to split them across multiple people?

Do you have a specific task in mind?

[mtrmac]

The way I’m currently thinking about things (without looking at much of the code), I’m planning to work on two somewhat independent aspects:

• Sharing the “search for unqualified images” code across projects (while perhaps still retaining project-specific heuristics)
• Sharing a “local images” wrapper on top of c/storage, with more high-level API. (I’m not at all clear on details about this one yet, I’m going to look at what the various projects do. This might well be split into several tasks, OTOH the document’s functionality as used by CRI-O is not _that_ big.)

Looking at https://docs.google.com/document/d/1GKub_NvdGCo7veR8XXf5w3PaEpIwzU6TnTMKNDcErEA/edit , it has another section about the “runtime service” (dealing with pods/containers), so that would be another set of tasks, not sure whether one or more.

[saschagrunert]

Do you have a specific task in mind?

Everything where you need help. 🙈

Looking at https://docs.google.com/document/d/1GKub_NvdGCo7veR8XXf5w3PaEpIwzU6TnTMKNDcErEA/edit , it has another section about the “runtime service” (dealing with pods/containers), so that would be another set of tasks, not sure whether one or more.

I’m not sure if the document is up-to-date since it has not been touched after the November meeting.

[vrothberg]

For the moment, I prefer to wait until @mtrmac 's draft for the new image package.

A somehow related big chunk of work is the new v2 API where we could need helping hands. Once that's finished, we could start tackling separating Podman and libpod. Doing that in parallel could cause to much noise.

@baude is the chief of the new API. @baude, do you have an endpoint/task that @saschagrunert could tackle? I can happily show how the new API works and how to test it etc. in a Bluejeans session (we're both based in Germany).

[vrothberg]

@saschagrunert, if you have cycles to work on the V2 API, the /libpod/image/$name/tree endpoint is no implemented yet (see https://github.com/containers/libpod/blob/master/pkg/api/handlers/libpod/images.go#L40).

The problem with tree is that most of it is done outside of libpod/libpod and we want to avoid code duplication by all means. So parts of the work would go into refactoring the tree code into something that can be used by both, the "native" podman client and the remote endpoint.

[saschagrunert]

@saschagrunert, if you have cycles to work on the V2 API, the /libpod/image/$name/tree endpoint is no implemented yet (see https://github.com/containers/libpod/blob/master/pkg/api/handlers/libpod/images.go#L40).

The problem with tree is that most of it is done outside of libpod/libpod and we want to avoid code duplication by all means. So parts of the work would go into refactoring the tree code into something that can be used by both, the "native" podman client and the remote endpoint.

Thanks, working on this one in https://github.com/containers/libpod/pull/5093