Sig-release: Automate deb/rpm publishing

[luxas]

This is an action item issue to make sure anago takes care of pushing debs/rpms in v1.8
Now it's a really painful process to have to ping various Googlers to push the packages manually

The proposed way of fixing this is by making a new GCS bucket that is set up as a deb/rpm repo.
It must maintain the structure of the current apt.kubernetes.io repo.

Then, in the release process, packages will be built and published by just pushing to GCS.

It's also a lot easier to manage ACLs with a GCS bucket compared to the Google-internal stuff we're currently depending on. (Where the source code/tools used are confidential)

For converting .deb packages to something pushable to GCS we can use https://github.com/spotify/debify
I don't know what's the best way to create a yum repo, but will look for something useful

cc @david-mcmahon @ixdy @jdumars @mikedanese @pipejakob @wojtek-t

[mikedanese]

Who is going to work on this?

[luxas]

Any volunteers?
cc @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-release-feature-requests @kubernetes/sig-testing-feature-requests

[timothysc]

So here is my conundrum, we need a couple of people to own this, with at least one person being a non-googler... and by own, I mean they will maintain it across at least 2 releases to ensure the automation is better than what we do today.

[ncdc]

I don't know what's the best way to create a yum repo, but will look for something useful

createrepo (see e.g. http://www.rightbrainnetworks.com/blog/using-amazon-s3-as-a-hosted-yum-repository/).

[jdumars]

It appears we do not have anyone stepping up on this. What is the best way to widen the net? These deliverables are an important part of the product we deliver during releases.

[ncdc]

I can try to work on the rpm side of things.

[mikedanese]

I agree with @timothysc's assesment. There's some significant unglamorous legwork that needs to get done here. I'll speak on Debs since RPMs seem to be covered.

Initially and potentially until the project stabilizes we will need to host our own apt repos. Reason for this: Debian self defines their stable channel as containing stable and well tested software, which changes only if major security or usability fixes are incorporated. They are very strict about this. If we got kubeadm in debian 9, it would have to be the same kubeadm version forever. We could only bump the version when debian 10 is released.

I'd like to bring deb build into our make release but that's gated on multi-platform bazel builds which is O(week or two) away. We could do the same with rpms but I don't know if that is desirable. At that point we will be able to publish raw debs to GCS. Then (a pretty big then) someone needs to build the publishing/signing pipeline.

[mikedanese]

I looked at https://www.aptly.info/ once upon a time. Might be useful for debs.

[mikedanese]

Also note:

There are currently 10 CNCF projects. Many of them have similar needs to us as far as OS packaging solutions. Let's see if we can kill 10 birds with one stone.

[timothysc]

Ideally with the maturing of atomic installation we could create a single container for *RPM installations backed this mechanism. The idea of curating our own RPM channel on this side-jiggery is odd.

/cc @jasonbrooks

[jdumars]

Is there any remote chance we could test automating these artifacts in the upcoming v1.8.0-alpha.3 release on 8/23?

[timothysc]

We need someone who is going to own maintenance of the packaging and I can work with them on it.

[spiffxp]

A reminder that we once again had to rely on someone cutting these manually for v1.8.0

@abgworrall did .debs, @pipejakob did .rpms

[spiffxp]

ref: https://github.com/kubernetes/release/issues/358

[timothysc]

I planned on talking with @pipejakob in more detail on the next sig-cluster-lifecycle mtg about this.

[jdumars]

Any chance this will get traction for 1.9?

[enisoc]

I recently started getting familiar with the internal docs for this process, and it currently requires Google-internal commands. That likely means we'd have to port it to public infrastructure as a prerequisite to automating the steps in an open-source tool (anago). @pipejakob can correct me if I'm wrong.

As a middle ground, I'm going to see about automating via an internal script, and/or spreading around expertise in this process. I also want to clarify who's responsible for publishing the packages. Ideally it should be the branch manager, but that wasn't the case when I was the branch manager for 1.6. @jdumars Who did it for 1.8?

[pipejakob]

@abgworrall and myself split the debs/rpms for the 1.8.0 release for expediency.

[enisoc]

@pipejakob Do you think it's too much work for one person? Or is it just that you wanted to minimize latency since it happens after the release already went out?

For 1.9, I'd like to absorb responsibility for pushing debs/rpms into the official release team (i.e. the branch manager, @dashpole) so it's under our full control, although we'll need help learning from those who've done it before. Does anyone have objections to that?

[timothysc]

@enisoc Ideally I want every build to look-like a release, except someone gets to push to the stable repo. I'd like to talk about this topic if it's an agenda item for sig-release.

[david-mcmahon]

@timothysc We are slowly moving toward something like that model with triggered staged releases. Not every build, but builds that pass the "release blocking" test set being staged and ready to release.

[enisoc]

I've now got a prototype for the "middle ground" automation I mentioned above in https://github.com/kubernetes/sig-release/issues/10#issuecomment-337297378.

This at least reduces the push process to one command per release, but it still relies on Google-internal tools. If you're a patch manager who's interested in trying it, let me know.

[luxas]

Great, thanks @enisoc! Is that coming as a PR to kubernetes/release or is it internally only?

[enisoc]

Unfortunately the script itself is also internal-only since it runs internal-only commands. This doesn't get us any closer to a process that non-Googlers can run, but it should help us get packages out more promptly in the meantime.

[luxas]

@enisoc No worries, thanks for improving the internal process meanwhile, will greatly reduce the time-to-pkg-publish metric :)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[jdumars]

/remove-lifecycle rotten

[jdumars]

/sig release
/sig cluster-lifecycle

[luxas]

Has there been any progress in automating this in anago?
With https://github.com/kubernetes/kubeadm/pull/803 & https://github.com/kubernetes/test-infra/pull/8020 I'm improving an e2e test that we'll put in all branch-specific sig-release-1.X-all dashboards (like: https://k8s-testgrid.appspot.com/sig-release-1.10-all) and the master-blocking tab (https://k8s-testgrid.appspot.com/sig-release-master-blocking), for more visibility in case someone forgets to push the packages.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

[justaugustus]

/lifecycle frozen

[spiffxp]

@timothysc We chatted about this in a couple of different contexts at KubeCon Seattle, does SIG Cluster Lifecycle have resources to commit to this for the v1.14 release cycle?

[timothysc]

Yes, the tracking issue for execution is here - https://github.com/kubernetes/kubernetes/issues/71677

We will need to PR here to remove some of the behavior.

[justaugustus]

/priority important-soon
/area release-eng
/milestone v1.15

[justaugustus]

Closing in favor of https://github.com/kubernetes/release/issues/631
/close

[k8s-ci-robot]

@justaugustus: Closing this issue.

In response to this:

Closing in favor of https://github.com/kubernetes/release/issues/631
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.