Cri-o: ci/prow/images and ci/prow/e2e-aws fail consistently

[kolyshkin]

Description

Currently, CI seems to fail consistently on the following jobs:

• ci/prow/images
• ci/prow/e2e-aws

The failure in both jobs is

http://vault.centos.org/centos/7/updates/Source/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
[ERROR] hack/build-rpms.sh:31: yum-builddep -y "${OS_RPM_SPECFILE}" exited with status 1.

(for complete log, see e.g. https://storage.googleapis.com/origin-ci-test/pr-logs/pull/cri-o_cri-o/4350/pull-ci-cri-o-cri-o-master-e2e-aws/1327016934725652480/build-log.txt)

and it is coming from here:

https://github.com/cri-o/cri-o/blob/48ee902491e35452801361a09940a342d0583537/hack/build-rpms.sh#L31

The failure seems to be related to the fact that vault.centos.org removed repodata directory from http://vault.centos.org/centos/7/updates/Source/ (note it's still available on a mirror, e.g. https://archive.kernel.org/centos-vault/centos/7/updates/Source/repodata/).

More importantly, it's not quite clear why we're still using centos 7.

Steps to reproduce the issue:

1. create a PR against cri-o/cri-o repo

Describe the results you received:

see above

Describe the results you expected:

ci/prow/* tests pass

Additional information you deem important (e.g. issue happens only occasionally):

Output of crio --version:

(paste your output here)

Additional environment details (AWS, VirtualBox, physical, etc.):

[kolyshkin]

@haircommander @saschagrunert PTAL (I think the source config for this CI comes from https://github.com/openshift/release/blob/master/ci-operator/config/cri-o/cri-o/cri-o-cri-o-master.yaml) and we need to use centos 8 maybe?

[kolyshkin]

found here: https://github.com/cri-o/cri-o/pull/4366#issuecomment-726385058

[saschagrunert]

Yes, I'm in favor of updating the distributions, but there may be other blockers/reasons behind that.

[fidencio]

Just for the record, let me add the info here as well.

The issue faced was in the CentOS Infra side, due to the 7.9.2009 release and I've filedI've opened https://pagure.io/centos-infra/issue/156 to track its status.

Fabian Arrotin promptly replied and worked the issue around in a way it unblocks us:
"Hi,

Some just reported the same issue on centos-devel list, and it seems that @hughesjr indeed forgot to update metadata. (and also some real src.rpm should also have landed there, probably still in his todo list for post-GA activities)

I've created empty repodata for now, so that shouldn't "block" you but I'll let this ticket assigned so that he can confirm and close it when properly implemented"

I've verified the workaround indeed unblocks us by triggering a /retest on #4356, which already passed the problematic part.

[kolyshkin]

I guess we can close it for now (and maybe plan to upgrade to centos 8 later)