Pipeline: Add Cluster scope pipeline support

[zhangtbj]

Expected Behavior

We can also define Cluster scope pipeline as ClusterTask

Right now, Tekton has Cluster scope task ClusterTask, but If I also want to run a Cluster scope pipeline which include that ClusterTask.

So that people can define Cluster scope pipeline for all cluster-level users

Actual Behavior

Something like:

apiVersion: tekton.dev/v1alpha1
kind: ClusterPipeline
metadata:
  name: clusterpipeline
  name: demo-pipeline
spec:
  resources:
  - name: source-repo
    type: git
  tasks:
  - name: skaffold-unit-tests
    taskRef:
      name: unit-tests
...

[vdemeester]

/kind feature
/kind question

[tomgeorge]

If nobody is working on this, I will start this evening.

[ImJasonH]

I believe there was a question about whether ClusterPipelines would be allowed to contain namespaced Tasks when run in a namespace. I think this is probably confusing enough we should punt on it until someone has a specific use case, and only support ClusterTasks in ClusterPipelines at first.

[tomgeorge]

got it, thanks!

[zhangtbj]

I think ClusterTask is just for some sample usage.

If it is an advantage or complex task or job, we need to define a pipeline to do more steps.

Such as cluster scope or level operation by using on the cluster level.

Or as we are in a multiple-tenants environment, we have a cluster scope pipeline that allows all namespaces multiple-tenants can use this cluster-level pipeline together. Or we have to define the pipeline in each namespace one by one.

[dbazhal]

Here is our case to use ClusterPipelines.

We're making multitenant system, each team has it's own project in "control" cluster. Each project has multiple environments(namespaces) like testing, prod, etc, in multiple clusters.
So, each service in each project should have pipeline that builds artifact, and delivers it promoting from environment to environment to production unchanged(because what we test on one environment should get the same as tested to production).
So what does this CI/CD pipeline? Launches some "build" pipeline. Then for each namespace in testing environment launches "deploy" pipeline. Then for each namespace in production environment launches same "deploy" pipeline.
Every service pipeline in every project uses that "build" and "deploy" pipelines.
And when the same "build"/"deploy" pipeline should be used in different namespaces, we want to code, test, and publish that pipelines once for all users, and make it available as ClusterPipeline.

In such case i agree that ClusterPipeline should be able to refer only to ClusterTasks, but not namespaced Tasks.
And maybe, if https://github.com/tektoncd/pipeline/issues/2134 will be done, ClusterPipeline could refer to other ClusterPipelines.

[jsimao71]

Support for cluster scoped Pipeline is really critical. A cluster will typically have many namespaces being created (in multi-tenant or even single-tenant), while only a small number of pipelines. Required the commonly used pipeline to have to be deployed in all such namespaces is a big burden on the use of tekon, and a waste of resources. Not to mention upgrading pipelines across all namespaces. Please add this...My believe is that this is a critical practical features for the ease of use and adoption of Tekton. Ideally a admin should setup common pipelines at cluster level, and all devs using the cluster can benefit from those pipelines.
Much appreciated..

[jsimao71]

BTW, the issue of the scope of tasks in cluster level pipeline is in concept easy to deal with.
A enum property can fine-tune the behavior. e.g.
task-scope: cluster-only | namespace-only | cluster-first | namespace-first
with the following meaning:
cluster-only -- use always cluster tasks (which need to be setup by admin)
namespace-only -- assume relevant tasks have been added to namespace
cluster-first [not as useful] -- check cluster task first and use if it exists, otherwise check namespace
namespace-first -- check namespace task (if defined), fallback to cluster task
The default could be namespace-first (but other are also reasonable).
Much appreciated!!!!

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[dbazhal]

na-ah, please don't
/remove-lifecycle stale

[jsimao71]

@dbazhal can you plese clarify why you think cluster support for pipelines is not a good idea?! Thanks!
Personally..I find really bad pipelines can not be installed at cluster level like task.
For exemple, if you create a simple but very common and useful pipeline with two task to git clone and build with maven and push image, this limitation of tekton implies that the pipeline needs to be installed in all namespaces.
Fundamentally, there is not much difference between pipelines and tasks. A pipeline of two tasks is almost a task, just a composite. Why is task support at cluster level, but not pipeline?!? Does not make much sense to me.
It really ruins the tekton experience, because a cluster admin can not setup tekton for easy install in all namespaces.
As it is requires all devs that own a namespace to reinstall all relevant pipelines.
So I insist please consider adding this feature.
And if not, please provide some rationale why not...
Thanks.

[dbazhal]

wat :) i was talking to bot that tried to stale issue.

[zhangtbj]

Yes, Our service leverages Tekton a lot.

And our service is not used for personal user, it is a multiple tenant case, it means we provide some shared Tekton tasks as ClusterTask, so that each namespace user can share or use them.

So it is the same case for Cluster scope pipeline. Such as for a common CI/CD or build relate processes.

[dbazhal]

We are currently compelled to copy each pipeline into user namespace to execute :D
That kinda says something about importance of cluster-scoped pipelines :)
I think feature importance is not an issue, there must be some trouble with adding this functionality. I'd like to help but i don't know how

[bobcatfish]

Hey all!

I think it is likely we'll be moving away from cluster scoped resources ( not sure if @vdemeester and others might disagree) as we add more support for tekton bundles (more tools and support coming soon!)

The idea is that there are a lot of limitations to referring to Tasks and Pipelines in the cluster (including namespace scoping and it being hard to version them (#1839)) so we are moving toward a model where you can reference Tasks and Pipelines other places, e.g. the feature we recently added allows you to store them in OCI registries (like images) and reference them there, and we'll likely add support for other locations such as git in the future.

[dbazhal]

Aw. Talking about solving concrete problem - absence of reusable pipelines mechanism - how soon could it be done? Wouldn't it be faster to add cluster-scoped pipelines first, and then calm and move to bright future with bundles?) (namespaced pipelines hurt really much)

[bobcatfish]

I think you can start using it today but I don't know what the mechanics of that look like b/c I think you'd have to create the Bundle by hand? Not 100% sure but I think @ImJasonH @vdemeester know more

[jsimao71]

tekton bundles makes sense if you could reference directly in taskruns and pipelineruns. If the can only be used to define pipeline specs -- as the docs suggest -- it does does solve the problem, because it requires the pipeline to be defined in each namespace again -- which is the problem at hand.
Even with bundles, one cloud argue that bundles are really a complimentary mechanism. It also adds extra complexity related with authentication in remote OCI registy.
Moreover if you don't add cluster level pipelines, and still keep cluster level tasks, the runtime becomes completely asymmetric. On the other hand, if you remove cluster tasks you loose backward compatibility of the model. Kubernetes has other native resources that are scoped at namespace and cluster level -- e.g. roles and role bindings. So it makes sense for tekton to maintain the spirit of K8s model.
The question is: how hard can it be to add pipelines at cluster level, for somebody that is familiar with the code?!?
If you think this is really hard to do, please provide some explanation why this is the case..otherwise the issue will continue to popup, people get confused, and you have lots of upset users.

[vdemeester]

tekton bundles makes sense if you could reference directly in taskruns and pipelineruns.

This is the case (https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#specifying-the-target-pipeline). You can use "reference" to tekton bundles in PipelineRun, Pipeline and TaskRun :wink:

That said, I do not think we should move away from cluster scoped resources, and the more I think about, the more I don't see why we wouldn't have ClusterPipeline.

The question is: how hard can it be to add pipelines at cluster level, for somebody that is familiar with the code?!?
If you think this is really hard to do, please provide some explanation why this is the case..otherwise the issue will continue to popup, people get confused, and you have lots of upset users.

This is not really the question. The code is a "detail", what is important is "does ClusterPipeline solve a problem for users ?" or does it make sense. I think it makes sense and as @jsimao71 @dbazhal @zhangtbj wrote, I think there is some need for it even. In OpenShift, having ClusterPipeline would simplify even more our "first time user" flows.

One problem to think about is @ImJasonH's comment

I believe there was a question about whether ClusterPipelines would be allowed to contain namespaced Tasks when run in a namespace. I think this is probably confusing enough we should punt on it until someone has a specific use case, and only support ClusterTasks in ClusterPipelines at first.

Should we allow only ClusterTask and bundle reference from ClusterPipeline ? If we do, what is the impact / confusion on the user ? If that's the case, it would mean, for example, that converting a Pipeline into a ClusterPipeline is more involved than changing the type (because you would also need to ensure tasks referenced are ClusterTask.

Anyway, I think it would make sense to discuss this and implement if we reach a consensus :wink:. It would also need to pass through a TEP.

[vdemeester]

I think you can start using it today but I don't know what the mechanics of that look like b/c I think you'd have to create the Bundle by hand? Not 100% sure but I think @ImJasonH @vdemeester know more

You can use oci for now, until we integrate this more into "official" tooling.

[jsimao71]

Thanks for the update and discussion @vdemeester.
As you point out, I think this thread shows clearly that lots of people in the community think this is a great idea.
This simplifies life for first time users --- which can really on high-level tools or clusters pre-setup by an admin). And also saves possible duplicate installs across namespaces, which is wasteful at best and a pain to maintain and update in the worst case.

On cluster scoped pipelines refering to namespace scoped tasks as you point out, this can lead to confusions and is somewhat error prone. The exception is when it works kind of "inheritance with override" -- that is, there is a cluster scoped task defined, but it is allowed for a namespace scope task with same name to override the cluster one.
Note, however, that because (cluster)tasks can have parameters, most uses cases where this "inheritance with override" could be considered, probably can be implemented more simply setting parameters in the task when running a task run.
So I suggest that, at least in a first iteration, not to support cluster pipelines references to namespace tasks.
When we have a stable implementation of cluster pipelines, and people start to use it, it might become clear if there are some valid use cases for references of type cluster pipeline->namespace task.
Thanks everybody for the comments...specially the "heroes" that are going to do the code implementation ;).