Cri-o: CentOS packaging strategy causes maintenance issues.

[matthias50]

Description

Packaging strategy for CentOS causes multiple issues such as bootstraps which used to work suddenly failing and unneeded yum config changes when updating cri-o.

Steps to reproduce the issue:

1. Configure yum to point to RPM repos for CentOS. For example https://cbs.centos.org/repos/paas7-crio-115-release/x86_64/os/

Describe the results you received:
I can only install a single version of cri-o, currently 1.15.1

Describe the results you expected:
I should be able to install any released version of cri-o starting at some "beginning of time" marker, which at least includes all 1.14.x and 1.15.x releases.

Once a version appears in a release repo, it should never disappear as this causes node bootstraps to suddenly fail if they lock down the version.

Output of crio --version:

N/A

Additional environment details (AWS, VirtualBox, physical, etc.):

CentOS 7

[mrunalp]

@lsm5 ptal

[lsm5]

Description

Packaging strategy for CentOS causes multiple issues such as bootstraps which used to work suddenly failing and unneeded yum config changes when updating cri-o.

The cri-o builds have been segregated this way for quite sometime now. Did you notice issues only recently? Could you please give more details of what exactly broke?

Steps to reproduce the issue:

1. Configure yum to point to RPM repos for CentOS. For example https://cbs.centos.org/repos/paas7-crio-115-release/x86_64/os/

Describe the results you received:
I can only install a single version of cri-o, currently 1.15.1

The CentOS PaaS SIG provides different kubernetes versions on different repos as well. That's why the separation here. This is totally intentional. Are you using kubernetes/openshift as well? Mind sharing details of how you install it?

Describe the results you expected:
I should be able to install any released version of cri-o starting at some "beginning of time" marker, which at least includes all 1.14.x and 1.15.x releases.

I did try dumping all different versions of cri-o on a single yum repo on CentOS CBS, but that didn't work as expected. Only the latest build would show up on the repo and kick other builds out.

Once a version appears in a release repo, it should never disappear as this causes node bootstraps to suddenly fail if they lock down the version.

AFAIK, upstream only supports the latest patch release of any version. So, if 1.15.1 is out, upstream doesn't support 1.15.0, and what's happening on CentOS is in sync with upstream. @haircommander @mrunalp correct me if I'm wrong.

Output of crio --version:

N/A

Additional environment details (AWS, VirtualBox, physical, etc.):

CentOS 7

[matthias50]

AFAIK, upstream only supports the latest patch release of any version. So, if 1.15.1 is out, upstream doesn't support 1.15.0, and what's happening on CentOS is in sync with upstream.

We are deploying upstream k8s using custom automation based on terraform and puppet. On node bootstrap, we install RPMs which have their version locked for all components which we install. That way we can first deploy updates in a non-production environment and then graduate them to production.

As an operator, I shouldn't be forced to always install the latest version. I should be able to perform my own vetting and then promote. With only a single version in the repo, it makes this harder to do. We essentially have to mirror the RPMs somewhere else so that they won't randomly disappear and break production when a new version comes out.

For kubelet and kubectl, we use the repo http://yum.kubernetes.io/repos/kubernetes-el7-x86_64. In this repo, you can find RPMs for all k8s versions going all the way back to 1.5.4. My preference would be that crio where managed in the same way, a single repo with all versions in it from which I can pick the one I want.

[matthias50]

I found an example of a multiple version of the same RPM being used for OpenShift Origin. If you look at https://wiki.centos.org/SpecialInterestGroup/PaaS/OpenShift-Targets-Tags-Repos, it talks about the process, then if you look at https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin39/, you can see multiple versions of the same RPMs all in the same repo. So it looks like this should be possible using CBS.

Can this be mimicked for a repo which just has cri-o in it?

[DanyC97]

@matthias50 few things to note...

https://cbs.centos.org in general should not be consumed publicly in the same was you would do with mirror.c.o. the endpoint is a yum repo but only use case is for _developer_ space to try things out.

buildlogs.c.o is for testing before packages are promoted to mirror.c.o (signed too).

Regarding openshift-origin39 vs paas7-crio-115-release there are totally different things.

Saying that i understand your ask, give us (@lsm5 and i) few days to have a chat with CentOS infra folks and we'll come back with an answer.

[lsm5]

@matthias50 few things to note...

https://cbs.centos.org in general should not be consumed publicly in the same was you would do with mirror.c.o. the endpoint is a yum repo but only use case is for _developer_ space to try things out.

This is my bad in that I've yet to make sure buildlogs correctly has the repos setup for cri-o.

[Prodian0013]

@lsm5 @DanyC97 are there any plans to have an official crio yum repo or use another repo such as epel? This is also related to #2826 it seems like there are several missing versions of crio on cbs.centos

[lsm5]

@Prodian0013 unlikely, cri-o will never be in base CentOS and epel won't allow setting up multiple repos for separate cri-o versions afaik. RE: cri-o version missing on cbs, I'll take care of that asap.

[matthias50]

@lsm5 Have you had a chance to get the buildlogs setup done?

[lsm5]

Not yet, been severely distracted. I'll try and get on it asap but I'd really appreciate if we could get some volunteers for CentOS tasks. Now with CentOS8 midstream stuff happening soon, I'm not sure how much more work that would end up being.

[lsm5]

@matthias50 i'm aware it's not convenient, but atm you can grab older builds from https://cbs.centos.org/koji/packageinfo?packageID=5695 if necessary.

[matthias50]

@lsm5, I'd be willing to lend a hand to expedite getting the buildlogs setup done, but I'm really not sure where to start. Can you point me in the right direction?

[lsm5]

@lsm5, I'd be willing to lend a hand to expedite getting the buildlogs setup done, but I'm really not sure where to start. Can you point me in the right direction?

Thanks, I guess first steps would be to create accounts on https://bugs.centos.org and https://accounts.centos.org (don't think it's single sign on). Would also help to be on #centos-devel on freenode. Let me know once that's done.

[matthias50]

Let me know once that's done.

Done

[matthias50]

@lsm5 suggested to me on freenode that I create a bug on the centos side for this. So I created https://bugs.centos.org/view.php?id=16566. @lsm5 and @DanyC97 did I phrase the request properly? Should there be a separate request to handle promotion to mirror.c.o?

[matthias50]

After some digging, it looks like the process is to create a pull request on the cbs-content-control repo. I opened this request: https://git.centos.org/centos/cbs-content-control/pull-request/13

[matthias50]

@lsm5 , can you comment on https://git.centos.org/centos/cbs-content-control/pull-request/13? It looks that needs to happen before it will get merged.

[matthias50]

For those of you following along with this issue, the requests are approved which should result in CRIO being published to buildlogs.centos.org with this commit: https://git.centos.org/centos/cbs-content-control/c/1f5843b9e6690d2307b0d1b98f72073ca5d17d1b?branch=master.

Once the CentOS infrastructure folks figure out why the sync is busted (and presuming the mirroring to buildlogs looks right), the next step is to make another commit to https://git.centos.org/centos/cbs-content-control to publish signed RPMs to mirrors.centos.org.

After all this is done, then all versions of RPMs should be available in stable repos and not just the latest version.

[afbjorklund]

@matthias50 : this is still a couple more months off, right ?

Don't _really_ want to mess around with the Nix binaries, if there will be an official RPM later on...

https://github.com/cri-o/cri-o/pull/2870#issuecomment-538624646

[matthias50]

I honestly wish I knew what the holdup is. For some reason, the mirroring isn't working and their are no publicly available logs for me to troubleshoot, so I'm left pestering people on IRC to look into it. @lsm5, is there anything you can do to help get this unstuck?

[afbjorklund]

Seems like there is no interest from downstream distributions (centos, ubuntu) to include this in the main repositories, for various reasons. And the current CBS and PPA are not always working either.

So for this (cri-o) to be a drop-in replacement, one probably has to supply third-party packages. So that one could download them from an upstream website, similar to how the current runtime works.

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

Maybe these could be hosted at https://cri-o.io ? Similar to https://containerd.io

[lsm5]

I honestly wish I knew what the holdup is. For some reason, the mirroring isn't working and their are no publicly available logs for me to troubleshoot, so I'm left pestering people on IRC to look into it. @lsm5, is there anything you can do to help get this unstuck?

Don't have anything apart from the nag them until they respond approach.

[matthias50]

@lsm5 in a related note, I don't see any 1.16 crio builds on cbs. Are these going to appear there or are the CentOS packages 1.16 going to be somewhere else?

[afbjorklund]

The https://cri-o.io/ documentation still features paas7-crio-311-candidate which uses 1.11 (Jun 2018)
As there are _lots_ of configuration changes between cri-o versions, all versions need to be provided...

https://cbs.centos.org/repos/paas7-crio-311-candidate/x86_64/os/

But it would be nice if the link could be updated to at least 1.15, while waiting for 1.17 to appear ? That is the version that the kubernetes documentation (kubeadm) is using, so hopefully it would be alright to use.

https://cbs.centos.org/repos/paas7-crio-115-release/x86_64/os/

---

Will there be a "paas8" for CentOS 8, or how does that work ?

Do you have to use CoreOS in order to run CRI-O on CentOS ?

[afbjorklund]

Will there be a "paas8" for CentOS 8, or how does that work ?

Do you have to use CoreOS in order to run CRI-O on CentOS ?

According to https://blog.centos.org/2020/01/centos-paas-sig-quarterly-report-2/
the CentOS SIG is providing Origin 3.x since before and will look at OKD 4.x next

So this means that CRI-O will be supported on CoreOS only, like upstream:

1. Single Node -> RHEL -> Podman. [...]
2. Multi Node -> OpenShift -> [RHEL] CoreOS -> CRI-O. [...]

There are no plans to offer CRI-O and CoreOS support separately from OpenShift [...]

There doesn't seem to be any technical reasons, assuming you can find a binary.

[matthias50]

Personally, I've resorted to getting the RPMs from the release directory under https://cbs.centos.org/kojifiles/packages/cri-o/. For example https://cbs.centos.org/kojifiles/packages/cri-o/1.15.2/1.el7/x86_64/cri-o-1.15.2-1.el7.x86_64.rpm. This is the "wrong" place to get them as they should be available on mirror.centos.org. I made a lot of effort to get these mirrored there and from what I understand, it should be working. Unfortunately, I don't know why it isn't and I don't have access to the error messages related to the problem and I can't get anyone who can see the messages to respond to me. :(. So I gave up.

[haircommander]

we now build for centos 7 on OBS, and as such have capacity to have the latest CRI-O release for upstream supported releases.
https://build.opensuse.org/package/show/devel:kubic:libcontainers:testing/cri-o-1.17
as such, I am closing this issue. If an issue pops up with OBS packaging, please reopen a new one.

[matthias50]

@haircommander, @lsm5 , the kubric stuff looks great. Once question I have left... When looking at https://github.com/cri-o/cri-o#getting-started, I see that repos are listed by version:

sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:[REQUIRED VERSION].repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:[REQUIRED VERSION]/CentOS_7/devel:kubic:libcontainers:stable:cri-o:[REQUIRED VERSION].repo

Are you planning on keeping all release versions of cri-o in there going forward, or just the latest on a release branch? Just want to confirm because my original issue was that if I use the upstream repos and pin to a specific patch version, then I could get into trouble if only one version (latest patch version) is present in a repo.

[haircommander]

@matthias50 we intend on supporting the three minor releases kubernetes does at the same time :smile: that's why they're different repos, so we can bump version older versions along with the newer :smile: does that answer your question?

@dougsland can I ask you to open PRs in github.com/cri-o/cri-o.io to update the packaging details?

[matthias50]

that's why they're different repos, so we can bump version older versions along with the newer 😄 does that answer your question?

I get that, but within each repo, will there be multiple patch versions? For example, in https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.15/CentOS_7/x86_64/, there is currently one rpm cri-o-1.15.3-7.2.el7.x86_64.rpm. When 1.15.4 is released, will there be a 1.15.4 rpm added to that repo, or will the 1.15.3 rpm disappear when 1.15.4 shows up? I would hope that once a rpm is released, it never disappears. That would allow us to pin specific patch versions (in this case 1.15.3) and not worry about the yum install randomly breaking when 1.15.4 is released. Then we can update our pinned version after we finish testing the new version.

[lsm5]

@matthias50 that hasn't changed yet. One repo continues to have just 1 build at a time.

The separation of repos was done so that users could do yum install cri-o without adding the version. https://github.com/cri-o/cri-o/issues/3495 . Hosting multiple patch releases isn't trivial on OBS and IIRC someone in the community (don't remember who atm) has some scripts running on their end to save older rpms. I'll share it soon as I can find out..

[lsm5]

Also, OBS folks mentioned about maintenance repo setup for keeping older versions. We would like more help in this, so if anyone in the community can volunteer toward this, that'd be awesome.

[lsm5]

@matthias50 a quick fix to support your need would be to create a subproject (and thus a new yum repo) for every patch release, so in case you wanna fall back on an older release, you can add the subproject repo. Would that work for you?

[matthias50]

@lsm5 . Thanks. This seems like it could be ok as an escape hatch to fall back. We've recently moved to building images which we promote through a CI/CD pipeline and staging environments instead of installing a pinned crio version during node bootstrap. So we may be able to let the crio image version float during image creation time.

One concern though. I noticed that there is one repo for the dependencies of crio (conmon, etc.) https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_7/devel:kubic:libcontainers:stable.repo and this is not versioned. Does this mean that no matter what crio version you install, the same versions of dependency packages are installed?

[lsm5]

One concern though. I noticed that there is one repo for the dependencies of crio (conmon, etc.) https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_7/devel:kubic:libcontainers:stable.repo and this is not versioned. Does this mean that no matter what crio version you install, the same versions of dependency packages are installed?

@matthias50 so the dependencies that are present in the main repo are also shipped (often older but thoroughly tested builds) in the default CentOS repos and the latter would be the recommended build to use in PROD. So, I'd suggest not using the main OBS repo if you want things to really not break. If that causes dependency issues for the cri-o packages in OBS, then do let us know and we can adjust those dependencies in the rpm sources