Pipeline: Feature: Versioning on Tasks/Pipelines

[pierretasci]

Abstract

A way to define a specific version of a named pipeline or task so that the spec of that pipeline or task can be referenced at a specific moment in time. For example, the way docker containers work today with tags whereby I am able to run container:tag for any valid tag of the same container name.

Use Cases

• In a production system, it is likely a project's Tekton pipeline will evolve its definition over time. If I wanted to run the pipeline as it existed a month ago, I have no good way to know what the spec was if it has since changed.

• The ability to test a change to a pipeline's spec without affecting the pipeline runs of other users that share the same spec.

• Improve the ability to share and reuse tasks so I can depend on a pinned version of a Task from a catalog while the actual spec changes.

Details

I could imagine this being addressed as part of the Tekton spec. For now, our workaround has been to publish tasks and pipelines with the "version" in the name of the task. This does introduce a discoverability problem as well as clutter.

A solution that is part of the Tekton spec would make a lot of sense. Eg:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: my-unique-tests
spec:
  - tag: v1
     spec: 
       steps:
         - name: run-test
            image: ubuntu
            command: ["foo"]
---
apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: my-unique-tests-run
spec:
  taskRef:
    name: my-unique-task
    tag: v1

An easy addition to this is a default "latest" to keep the existing functionality, again, similar to docker.

[vdemeester]

I think the main question is, is this in scope for tektoncd/pipeline or some higher level component (from Tekton or not) ? versionning could be done using different name and labels, … and managed by an higher level component, keeping the tekton api as simple (relatively speaking) as possible.

This is also where experiment like catalgos (expect some code in the next week :angel:), from https://github.com/tektoncd/community/issues/53) comes into place (on the discoverability side of things.

/area api
/kind question
/kind design

[pierretasci]

I completely agree that the main question comes down to whether this lives in tekton or externally. I think that both are viable. Since I brought up the issue, I will make a case in favor of making it a part of the Task spec and that is to aid in task reuse. If the goal is to make tasks that are generic and reusable with a well-defined interface, they will need to be versioned somehow to prevent the definition being changed while an old version is running.

A versioned catalog could work but acts as a coarse-grained lock in a sense. It might slow down changes if one of the tasks in the catalog is incompatible in the new version but you really need the changes introduced to another task.

[bobcatfish]

I agree - esp. when we start thinking about tektoncd/catalog. We'd want to be able to make sure we can make changes in there and folks can consume those changes as they want to.

@pierretasci I think we've also run into some complication around expressing the version of Tekton Pipelines that a Task is compatible with, do you see that as being a related problem or maybe totally independent? (or @vdemeester maybe this is going to be handled by v1beta1 v1beta2 etc. once we start actually incrementing those?)

kind: Task
metadata:
  name: my-unique-tests
spec:
  - tag: v1
     spec: 

It took me a couple min to realize that in this model we would need to have all the versions within one Task instance because otherwise their names would have to be distinct.

One downside is that the verison would become almost mandatory :thinking: or at least not using the version would look something like:

kind: Task
metadata:
  name: my-unique-tests
spec:
  - spec: # two specs for no reason :(

I wonder if we could brainstorm a few more options? One would be to embrace using the Task name as you mentioned - maybe it's reasonable to introduce a convention into the name for versions?

Ugh I can't think of much more... maybe by default we don't have spec.spec and we have something like:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: my-unique-tests
spec:
     steps: # this is the current version, so don't _have_ to use versions
       - name: run-test
          image: ubuntu
          command: ["foo"]
     versions: # you can optionally provide previous versions of the Task
    - tag: v1
       spec: 
         steps:
           - name: run-test
              image: ubuntu
              command: ["foo-old"]

I dunno tho, it seems like using latest causes a lot of problems, if only b/c you don't know what you ran... :weary:

Any other ideas?

[dlorenc]

Big +1 on solving this somehow.

I think I prefer to put the version info on a label of each Task, then we could add support for TaskRuns/Pipelines to include selectors in their TaskRefs.

This would work for things like "latest" as well as pinning to specific versions, but would not allow full semver-style version comparisons.

[pierretasci]

I really like the labels idea because 1. it is built-in to Kubernetes (as are selectors), and 2. it is open enough to allow anyone to change how they want to define their versioning. I wonder how we would handle reconciliation at the controller level when registering two tasks with the same name. AFAIK, the reconcile loop checks namespace and name for uniqueness.

Just to exhaust all possibilities, the other (horrible) idea I have is to have an inheritance chain a la kustomize. In this way, a task or pipeline specifies a parent task/pipeline and that spec is merged all the way down to a root CRD. This would be a nightmare to handle though (what if one piece of the chain is missing?).

To @bobcatfish's question, I think versioning Tekton itself goes hand in hand with with versioning the work being done by a task/pipeline. I may want to take advantage of a new tekton feature which has a new field in the spec but that may blow up in someone's cluster if they don't have the latest Tekton controller.

[bobcatfish]

Discussed this with @ImJasonH a bit and he has another idea that's pretty cool and very different from what we've discussed so far! This is something that he and a few other folks (@dlorenc @jonjohnsonjr ) have been throwing around as an idea as well, which has the potential to solve a few problems at once.

The preview is that we extend taskRef to support more stuff than just Tasks that live inside the cluster - and specifically we use OCI Artifacts to bundle and store Tasks, e.g.

apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: my-task-run
spec:
  taskRef:
    image:
      name: gcr.io/my/catalog:v1.2.3
      task: my-task

And we could even do cool stuff like:

apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: my-task-run
spec:
  taskRef:
    git:
      url: https://github.com/my/repo
      commit: deadbeef
      path: path/to/my/task.yaml

Jason's full proposal for Tekton Task References

[vdemeester]

I am definitely interested in re-using oci images/artifacts for this, a lot of tooling exists for it, and it handles versioning really well. I think we discussed it really early when talking about the catalog.

We could use labels for specifying stuff like "minimum version of tekton required", …

[bobcatfish]

@pierretasci do you have any thoughts/objections on this? no worries if you feel like you need more information first - I think this is pretty cool and am motivated to continue exploring this option

[pierretasci]

Sorry, I wrote my feedback in @ImJasonH's doc and forgot to post back here. I think the OCI proposal makes a ton of sense and I think it more than solves the use case intended here. I'm good to explore that as the proposal here.

[siamaksade]

cc @sthaha

[sthaha]

Doesn't using oci images make the definition of tasks opaque to user? Currently we can't treat tasks as opaque (akin to calling a function without knowing its implementation details) as in order to use a task, you must know the input and output params expected. In the OCI based implementation how would someone find out how to use the task?

[ImJasonH]

We can build tooling to describe a task defined in an OCI image, by cracking the image open and parsing the YAML. This could start as a standalone CLI in experimental then maybe eventually graduate to the tkn CLI. Something like tkn task describe my-task --image=gcr.io/my/image

[sthaha]

@bobcatfish I am a bit confused as to how using oci artifacts address the versioning issue? Hypothetically, say my pipeline refers to v1 and v2 of a task that run in parallel

• how would the spec look like?
• where would the tasks be created?
• are these tasks transient (deleted soon after its use?)

As I see it, the issue is that taskRef refers to the metadata.name of a Task thus we need to somehow encode version and name of the task into metadata.name. Rather how about we use labels to refer to a task? e.g.

taskRef:
   name: catalog.upstream.foobar
   version: 0.1.1

would use a task (independent of metadata.name) that has the label name and version

[sthaha]

@ImJasonH thanks for explanation. It does make sense to me and additionally addresses publishing a catalog issue. In the oci artifacts based proposal, would the controller even need to create tasks in cluster?

I am however not sure if this addresses versioning of tasks unless we can build a structure/naming convention into this oci artifact based catalog.

[ImJasonH]

This addresses versioning because OCI images can be versioned and referred to by their versions registry.com/image:v1.2.3, or pinned to a specific immutable version by its content (registry.com/image@sha256:abcde...).

In this model the Task definition doesn’t have to be defined in the cluster, the image only needs to be readable and reachable by the Tekton controller running on the cluster.

[ImJasonH]

The example from the doc shows how a taskRef might refer to an image, rather than a Task definition installed on the cluster:

apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: my-task-run
spec:
  taskRef:
    image:
      name: gcr.io/my/catalog:v1.2.3
      task: my-task

[sthaha]

This addresses versioning because OCI images can be versioned and referred to by their versions registry.com/image:v1.2.3

Neat! so here, it is the catalog that has a version which I guess all tasks inherit i.e. instead of specifying use task and version we specify task from catalog with version which is fine!

[ImJasonH]

Yeah, the image can contain a bunch of co-versioned Tasks/Pipelines/whatever, or just one each if the image author wants that. An image containing a bunch of co-versioned things can be considered like a Catalog.

[siamaksade]

Interesting idea. +1 for the proposal

@ImJasonH are there advantages to combing multiple tasks into a single image rather than enforcing a single task per image? If each image contains a single task and related step images, the syntax becomes simpler since the name is not needed anymore:

apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: my-task-run
spec:
  taskRef:
    image: gcr.io/maven:v1.2.3

This would be also somewhat similar to how GitHub Workflow references actions e.g. actions/setup-java@v1

[ImJasonH]

The advantage to co-versioning things together is if you have a Pipeline that runs multiple Tasks, you can co-version all of them together if they're all bundled in one image. But if you as an operator or thing author wants them separate, that's your prerogative.

GitHub's actions/setup-java@v1 thing is a bit more like how we use images as steps, since that reference ends up describing the container image that gets run as part of that step. With GitHub, that reference can also be a GitHub repo that might get built just-in-time into an image that runs.

[sthaha]

@ImJasonH One more question about the "opaqueness" of oci artifacts. Say we want to support a case where we would like to expose list of tasks in the catalog to a UI (dashboard). Would oci-image allow for that? Or would we have to rely on a Catalog CRD to expose that information?
e.g.

kind: Catalog
spec:
  imageRef:  gcr.io/my/catalog:v1.2.3 .  ## could be Refs instead

# filled by controller
status:
  tasks:
     - name: kaniko
     - name: buildah  
  clusterTasks:
     - name: ...

[ImJasonH]

You could ask the registry over HTTP for image metadata and use it to list the objects it contains. The API is well documented and part of the OCI spec, we would only be defining the spec for the data types in the image. We could easily provide a Go package to parse it all.

Having it wrapped in a CRD means there are two sources of truth: the CRD could say it contains A, B and C while the image itself contains A, C and Z.

[ImJasonH]

The OCI image also wouldn't have a concept of a ClusterTask since to the image there's no concept of a namespace or a cluster-scoped anything. If operators want to make sure some images _aren't_ available to some users they can enforce that with OPA and/or auth.

[siamaksade]

The advantage to co-versioning things together is if you have a Pipeline that runs multiple Tasks, you can co-version all of them together if they're all bundled in one image. But if you as an operator or thing author wants them separate, that's your prerogative.

GitHub's actions/setup-java@v1 thing is a bit more like how we use images as steps, since that reference ends up describing the container image that gets run as part of that step. With GitHub, that reference can also be a GitHub repo that might get built just-in-time into an image that runs.

For the user authoring pipelines, GitHub Actions and Tekton Tasks are similar in that they are components that user can reuse to build a pipeline. step is not a reusable component.

The runtime model of GitHub Actions is similar to a step like you said.

[pierretasci]

One small thing I want to add to this discussion that I really like about the idea of using OCI, is that auth is a "solved" problem. End-users will be able to publish their task and pipeline definitions as artifacts and use imagePullSecrets to authorize exclusively themselves to access those definitions in their CI. Not reinventing the wheel here is a huge win.

[ImJasonH]

Thought about this a bit more, this should be possible to prototype entirely in experimental, with a CLI that's able to bundle tasks/etc into an image, and a "run this task in this image" surface that resolves the Task spec from the image before telling Tekton about it.

Once this is iterated on and the surface settles, it should be easy to shift the image->Task resolution to the Tekton controller and move the bundling CLI into tkn.

[sthaha]

@ImJasonH @pierretasci Have you started working on this? It would be great I could pair or start the work on this.

[ImJasonH]

@ImJasonH @pierretasci Have you started working on this? It would be great I could pair or start the work on this.

I haven't, and I won’t have an opportunity to for a couple months. If you want to get started go ahead I think a prototype would be a great start and would help people iterate on the idea.

[pierretasci]

Hey sorry, I forgot to reply back to this thread. I spoke with Sunil and we are tag teaming on a prototype.

[sthaha]

Thanks to @pierretasci we have our first POC - https://github.com/tektoncd/experimental/pull/461

[bobcatfish]

@pierretasci has expanded on the proposed OCI format at https://docs.google.com/document/d/1lXF_SvLwl6OqqGy8JbpSXRj4hWJ6CSImlxlIl4V9rnM/edit

[pierretasci]

First PR with the new design is up https://github.com/tektoncd/pipeline/pull/2395

[sbwsg]

Moving to 0.18 release milestone because there are some remaining questions around fetching large images during reconcile.

[dibyom]

Closing since #2395 was merged.

/close

[tekton-robot]

@dibyom: Closing this issue.

In response to this:

Closing since #2395 was merged.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.