Test-infra: Proposal: Support git providers other than GitHub in prow

[wbrefvem]

What would you like to be added:

Some sort of generic git service in prow that normalizes incoming webhooks into a generic format that hook and other plugins can understand, as well as provides a generic interface for performing git-related actions such as commenting on PRs, etc.

My initial proposal is that this git service essentially be another plugin that implements a number of generic git-related APIs, and that the name/location of this service be configurable so that we can support out-of-tree providers. In addition, it's probably worthwhile to think about decomposing the notions of code review and issue tracking into separate provider APIs, since not all providers implement these functions, although presumably prow will only ever be useful in combination with a git provider that at least provides basic code review features. But it would be nice to support, e.g., code review tool X in combination with issue tracker Y. We currently have something like this with the loose coupling of the Gerrit adapter and reporter. It would be great if we could make this sort of thing generic and extensible.

We discussed this briefly on the sig-testing call of November 6, and my hope is that this issue will be an asynchronous continuation of that discussion. I am also working on a more formal proposal that I plan to present on the weekly sig-testing call some time in the near future.

Why is this needed:

Currently, prow is tightly coupled to GitHub. Making it more generic would give the various k8s projects more flexibility in how they craft their workflows, as well as making prow more accessible to projects in the broader open source community and beyond.

/kind feature

[BenTheElder]

cc @cjwagner @krzyzacy
/area prow

[yastij]

we might want to think about an interface that providers might want to implement.

[wbrefvem]

@yastij Yeah, there seems to be some consensus around this.

[wbrefvem]

Update

Here are some more concrete ideas on how to tackle this:

• Implement a git service as a freestanding gRPC server
• At least one webhook normalization method that would transform an incoming webhook into a generic event that hook can dispatch
• Individual methods for various actions such as AssignIssue, CommentOnPR, etc.
• Ship a default service that supports popular providers such as, but not necessarily limited to, GitHub, GitHub Enterprise, GitLab, Bitbucket.
• Make protobuf package and .proto files available for users who want to implement their own service.

[midnightconman]

This PR https://github.com/kubernetes/test-infra/pull/11075 was just merged, it gets us closer to this goal.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[yastij]

/remove-lifecycle stale
/lifecycle frozen

[townsendb]

Hi @BenTheElder . Is this proposal on the radar? I'm using GitLab and trying to use the next gen serverless features of Jenkins-x but blocked by this issue in prow.
Many thanks.

[McShauno]

Ok, fine ... I'll write it. Give me a couple of weeks.

¯_(ツ)_/¯

[BenTheElder]

@townsendb Hi, Sorry I missed this comment in all my github notifications somehow... I don't really work on prow currently (kind mainly) though I'd love to see this happen. I'm not sure where this is at.

@cjwagner & @stevekuznetsov are good current points of contact for prow.

[BenTheElder]

I think there was a more specific proposal related to this for the proposed "scalleywag" component but I'm not sure where that's at.

[polothy]

Just for those not aware, scallywag is here https://github.com/kubernetes/test-infra/pull/12241 and part of it was merged here https://github.com/kubernetes/test-infra/pull/12700

[berbiera]

Hi there, Is there a chance to get this proposal accomplished ?

[bgrant0607]

Previous request for gitlab: #8435

(connecting dots to make this issue easier to find)

[drhender]

I am also attempting to integrate prow with our internal deployment of Gerrit, but the limitations described in this issue prevent me from doing so.

To be clear, we use internal deployments of K8s and Gerrit-- not something deployed on a public cloud. Without support for such an environment, Prow (and Argo and Knative and Tilt and a whole host of other "K8s native" CI/CD tooling) is of limited use.

[BenTheElder]

@drhender it's worth noting that Prow unlike argo, knative, tilt etc. was originally written out of needs for the kubernetes upstream project to do it's own testing, rather than as a general tool. It was never expected to work in private environments and I think that's reasonable as opposed to knative, which hopefully does!

Now, Prow actually does have limited support for private Gerrit specifically at the moment, so it might be worth looking into that a bit more. I know this is actually in use.

[stevekuznetsov]

@drhender so the issue here is about supporting GitHub-like git providers like GitLab, IMO. Right now gerrit workflows do have support but since gerrit provides a _very_ different UI and workflow than the git/issue/tracking/whatever solutions like GitLab or GItHub, not every part of Prow is supported on gerrit. What issues are you running into?

[whiskerch]

Is there any update on this?
We are using prow on our projects, but need to set up some new ones on an isolated network - we're going to host our own gitlab instance as the cost of a self hosted github instance is prohibitive

[deedubs]

I'm interested in taking a look at adding gitea support where are we with regard to implementing @wbrefvem's suggestion of gRPC vs adding the additional services in tree

[RafalMaleska]

What is the status of this issue?

I was trying to use Lighthouse for Prow integration into Gitlab - it seams that it provides only basic prow functionality and requires JenkinsX installation.

Is there a RoadMap to develop Prow further?

[alvaroaleman]

What is the status of this issue?

If someone is interested in working on there will be progress.

Is there a RoadMap to develop Prow further?

No, prow is mostly developed by the ppl that use it according to their needs

[abdennour]

we love prow. But, we are using bitbucket cloud.
Any change ?

[cjwagner]

@abdennour No change, Alvaro's comment is still accurate. PRs are welcome if you'd like to contribute to this!