Pipeline: Pipeline Catalog Integration story

[vdemeester]

Now that we have tektoncd/catalog, that store re-usable task for now (and maybe more in the future), we may want to think of a way to use/integrate those in a pipeline flow.

That said we could provide functionality on Tekton side that help CI system is realizing this.
Something that might help is the ability for a task/pipelineRun to reference a task/pipeline by some kind of URI (instead of name only) e.g. git://repo/path/to/pipeline@gitref. A task/pipeline defined that way would be pulled from git and expanded in the task/pipelineRun definition. That would allow combining the power of embedding with the reuse of pipelines.

As @afrittoli said here, being able to refer directly to a catalog (there might be more than one, see openshift/pipelines-catalog for openshift's specific ones) would make easier for user to adopt pipelines.

This is also related to our Dogfooding effort and the follow pull-requests : https://github.com/tektoncd/catalog/pull/43 vs https://github.com/tektoncd/pipeline/pull/949 and https://github.com/tektoncd/pipeline/pull/954. As of today, we either need to have some script that apply the task from the catalog (kubectl apply -f https://…) or doing it manually prior to the pipelines/tasks executions.

Raw ideas

• Being able to use an url or a reference (à-la-container-image) in the TaskRef
• Using images to get the task definition, and thus being able to use image reference in the TaskRef. This would be a bit similar to what mattmoor/kontext does, or in a more "standard" way what docker/app and CNAB are doing too

Next steps

• [ ] Updating this issue based on comments and feedback
• [ ] Preparing a Design doc on this

[vincent-pli]

Could we leverage GUI (dashboard) or CLI to do this?
I think the import thing of Catalog is how to let user know there is stuff in catalog could be reused, no need another wheel.

So, if CLI or GUI could list all stuff in catalog and allow user apply some of them, this could help.

[afrittoli]

@vincent-pli I think having the ability of importing Tasks / Pipelines via CLI / GUI is a nice feature to have for sure; however it doesn't really solve the issues outlined in https://github.com/tektoncd/pipeline/issues/859 (see also my comment).

Being able to point to a Task / Pipeline by a URL of some kind would allow CI system to run pre-merge Task/Pipelines without polluting the namespace with copies.

Apart from the CI use case, a large organization could use this feature to give teams with tailored tasks and pipelines which are maintained in a central repo by a dedicated team. The CLI/GUI approach could be used for this use case as well, however teams would have to regularly check for changes in the central repo and orchestrate pulling in a new version of Tasks / Pipelines when available.

[vincent-pli]

@afrittoli

Being able to point to a Task / Pipeline by a URL of some kind would allow CI system to run pre-merge Task/Pipelines without polluting the namespace with copies.

Support URI or something similar in the Pipelinerun means we will not actually create related crd (tasks, pipeline, resource) in the k8s cluster, am I right?

But I guess there is challenge how to describe this to user, since we have Tasks, Pipeline, resource cannot to be a truly crd which could be trace or monitor by user 😄

[bobcatfish]

I may have duplicated this in https://github.com/tektoncd/catalog/issues/45

[vdemeester]

Design doc is https://docs.google.com/document/d/1O8VHZ-7tNuuRjPNjPfdo8bD--WDrkcz-lbtJ3P8Wugs/edit#

But I will close this one as a duplicate of tektoncd/catalog#45 :wink:

/close

[tekton-robot]

@vdemeester: Closing this issue.

In response to this:

Design doc is https://docs.google.com/document/d/1O8VHZ-7tNuuRjPNjPfdo8bD--WDrkcz-lbtJ3P8Wugs/edit#

But I will close this one as a duplicate of tektoncd/catalog#45 :wink:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.