Test-infra: Clarify audience of Prow

[zombiezen]

It's unclear to me from looking at the Prow documentation as to whether Prow is intended for use by projects other than Kubernetes. If so, I would expect the documentation to include more information about how to configure the tool for other workflows. If not, then I would expect the documentation to more clearly state that. A key part of my confusion stems from this sentence in the README:

Please note that these libraries are intended for use by prow only, and we do not make any attempt to preserve backwards compatibility.

[zombiezen]

(For context, we are evaluating merge bots for use in Go Cloud: google/go-cloud#687)

[stevekuznetsov]

/kind question
/cc @fejta @spiffxp

What "other workflows" are you talking about? There's nothing that ties Prow to the Kubernetes project and there is a list of other deployments for other projects in the docs.

[zombiezen]

What I'm missing here is I see a lot of documentation about how to deploy and write plugin code for Prow, but very little documentation for if I don't know exactly how Prow works, what things will it do to my issues when I deploy it, and how do I configure those things? If I am to set up Prow, beyond provisioning machines, what else do I need to do to have a successful Prow installation?

[stevekuznetsov]

Have you read the deployment docs ?

[zombiezen]

I have read the deployment docs. They do not answer the questions I have. That doc answers "how do I start running the service?", not "what does the service do once it is running?" nor "what controls (or jobs, but I don't quite understand what that is yet) do I have and how I discover this?".

I am not directly looking for answers on how to solve a particular problem right now; I am trying to give feedback on how I experienced the documentation. I do know how to deploy services with k8s, but I am a total novice to Prow.

In digging more, I see that there is a configuration README. I would expect this to be part of the "Getting Started" materials and have proper documentation of describing the default behavior of Prow etc. Instead, this doc is off to the side and then points directly to a Go struct deep in the innards.

[stevekuznetsov]

I'm a little lost ... there is a "features" section right at the top of the Prow README. Under the "more details" section there's a link to more docs and those link to the live self-hosted nitty-gritty details of "what will Prow do to my issue/PR" here. Could you maybe enumerate some concrete questions you have to help us write better documentation to cover those gaps?

[stevekuznetsov]

The questions I've gathered so far:

I would expect the documentation to include more information about how to configure the tool for other workflows

Without understanding what you mean by "other workflows" I don't really know how to help here.

what things will it do to my issues when I deploy it

Documented as mentioned in the previous comment

how do I configure those things?

If you are a job author for an exisitng Prow deployment, there's a document explaining that. If you're setting up a new deployment, there are a couple sections of the getting started/deployment guide that walk you through this.

If I am to set up Prow, beyond provisioning machines, what else do I need to do to have a successful Prow installation?

This is kind of a weird question -- if you follow the deployment guide, you will have a fully functional service.

If you've followed the deployment guide, you should have configured plugins, the infra, and written jobs -- those being the controls you have over how Prow runs.

I'm not sure we have had new adopters in the past who do not understand what Prow is or what a job is, since the growth has been organic from people exposed to the project. Do you think we'd be well served by adding to that features list a link to the configuration mechanism for each feature?

[BenTheElder]

This is a lot of text ^ 🙃 Couple of points:

Please note that these libraries are intended for use by prow only, and we do not make any attempt to preserve backwards compatibility.

Libraries != the binaries! We necessarily have some internal libraries for building prow, we don't make any particular effort to version those outside of prow.

_Prow_ (not the libraries) is definitely used by projects other than Kubernetes, the docs should have a list of users now. EG OpenShift, Knative, Istio, Cert-Manager, ...

The rest regarding setup etc should be covered by the getting started guide.

/cc @cjwagner @fejta

[fejta]

Prow will be useful to anyone who is familiar with kubernetes and can express their workload as a pod. Anyone familiar with golang should be able to learn how to extend it.

Agreed that our setup docs aren't the best (we aren't a startup trying to sell you something so this isn't what we optimize for). But we're slowly working on it!

[nikhita]

It's unclear to me from looking at the Prow documentation as to whether Prow is intended for use by projects other than Kubernetes

The prow README says Prow is a Kubernetes based CI/CD system. @zombiezen Do you think it would make it easier to understand on the first glance if we clarify it to say that Prow runs on Kubernetes but the projects using it don't necessarily have to be Kubernetes-related (phrased better of course)?

what things will it do to my issues when I deploy it

If you follow the getting_started_deploy.md, it just says that it will add a size/* label to your PR and not do anything fancy. :)

how do I configure those things?

There is a config.yaml and plugins.yaml file that defines basic behaviour for what Prow will do to your issues and PRs. To understand what these plugins and configurations _can_ do, I think we can maybe point to the contributor-playground repo? You can try slash commands - like /assign which will assign you to an issue/PR or /pony which will show you a picture of a cute pony - and see how they work.

It definitely won't reflect everything that prow does (jobs, etc) but it can give you a platform to try out plugins in action and decide which ones you'd like to use in your own Prow instance. Do you think this would help?

Once you see them in action and decide which ones you want to use, you can configure your instance by adding more plugins and jobs.

Like @stevekuznetsov mentioned, the Prow userbase has grown organically so it's generally people who have either already used Prow in some form or have been remotely related with the Kubernetes project, so they were already familiar with the slash commands and other prow-related things.

[zombiezen]

Thanks for the responses everyone. To @nikhita, replies inline:

It's unclear to me from looking at the Prow documentation as to whether Prow is intended for use by projects other than Kubernetes

The prow README says Prow is a Kubernetes based CI/CD system. @zombiezen Do you think it would make it easier to understand on the first glance if we clarify it to say that Prow runs on Kubernetes but the projects using it don't necessarily have to be Kubernetes-related (phrased better of course)?

The phrasing change you suggest would help, but I'm curious: does Prow actually run tests? For the usage I would be interested in, I would like to continue to use Travis for CI, and then use Prow as a submission mechanism. CI/CD does not communicate "automated change submission" to me coming from Travis.

what things will it do to my issues when I deploy it

If you follow the getting_started_deploy.md, it just says that it will add a size/* label to your PR and not do anything fancy. :)

Ah okay. It would be nice to state that earlier in the doc. As a new user, I would be afraid to install anything on a live GitHub repository without knowing its potential impact.

how do I configure those things?

There is a config.yaml and plugins.yaml file that defines basic behaviour for what Prow will do to your issues and PRs. To understand what these plugins and configurations _can_ do, I think we can maybe point to the contributor-playground repo? You can try slash commands - like /assign which will assign you to an issue/PR or /pony which will show you a picture of a cute pony - and see how they work.

It definitely won't reflect everything that prow does (jobs, etc) but it can give you a platform to try out plugins in action and decide which ones you'd like to use in your own Prow instance. Do you think this would help?

Yes! That would be great to have in the README as a "try it out/demo" sort of link.

Once you see them in action and decide which ones you want to use, you can configure your instance by adding more plugins and jobs.

Oh woah! I definitely did not find the plugins link when I was digging through docs. Even just having a list of possible commands is helpful in understanding (which I can get indirectly through that page).

Like @stevekuznetsov mentioned, the Prow userbase has grown organically so it's generally people who have either already used Prow in some form or have been remotely related with the Kubernetes project, so they were already familiar with the slash commands and other prow-related things.

Totally makes sense. Apologies if I came across as criticizing; my intent is to give feedback/new user perspective to help build out that documentation. I have made small fly-by contributions to the k8s ecosystem before, so I remembered other maintainers using Prow and thinking it was super cool. Unfortunately, that doesn't translate into a good mental model of the capabilities of the tool, so that's the gap I was hoping to fill in.

[stevekuznetsov]

I think the current getting started / deployment doc does walk through a lot of the questions that were raised but it's true that standing up a cluster and pointing it at a live GitHub repo is a large bar to jump over to get an intro -- we should do better docs to point people to the playground (I didn't even know it existed :eyes: ).

[nikhita]

Apologies if I came across as criticizing; my intent is to give feedback/new user perspective to help build out that documentation.

@zombiezen No worries, new user perspective is really helpful and appreciated! :)

but I'm curious: does Prow actually run tests?

Yes, you can define jobs in Prow. For example, this job defines presubmit jobs/tests to run on a PR in the kubernetes/community repo. It will checkout the code in the PR and then run make verify on it. The Makefile is in the repo itself. These tests will run in a pod on your cluster.

For the usage I would be interested in, I would like to continue to use Travis for CI, and then use Prow as a submission mechanism.

You could definitely do that as well, but fwiw prow offers you lots of extra benefits like re-testing using the /retest command and displaying the test results on a nice-looking dashboard: https://prow.k8s.io/.

I recently migrated from circleci to prow on an external cluster -- writings prow jobs isn't too much different than writing a travis/circleci config, especially if you have the tests in a script/makefile already.

we should do better docs to point people to the playground

Let me try to surface the playground repo + other links better. :)

/assign
Will take a dab tomorrow or so, if no one gets to it sooner.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta]

https://github.com/kubernetes/test-infra/blob/master/prow/getting_started_deploy.md#tackle-deployment will walk you through deploying prow into your cluster.
/close

[k8s-ci-robot]

@fejta: Closing this issue.

In response to this:

https://github.com/kubernetes/test-infra/blob/master/prow/getting_started_deploy.md#tackle-deployment will walk you through deploying prow into your cluster.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.