Pipeline: Allow embedding specs in Pipeline and PipelineRun instead of referencing.

[cjwagner]

Expected Behavior

I see that the TaskRun spec allow Tasks and PipelineResources to be embedded into the spec via the taskSpec and resourceSpec fields. (Docs, Issue) Creating just the TaskRun is simpler than creating all the CRDs and avoids the need to clean up ephemeral Tasks and PipelineResources that are only needed for the individual TaskRun. (ref https://github.com/tektoncd/pipeline/issues/544)

It would be great if PipelineRuns could expose a similar embedding mechanism so that the Tasks, Pipeline, and PipelineResources could all be embedded into the PipelineRun spec. Here is a simple example of what I would expect a fully embedded PipelineRun to look like (snipped together from other examples):

apiVersion: tekton.dev/v1alpha1
kind: PipelineRun
metadata:
  name: do-the-thing
spec:
  resources:
    - name: source-repo
      resourceSpec:
        type: git
        params:
          - name: url
            value: https://github.com/pivotal-nader-ziada/gohelloworld
  pipelineSpec:
    resources:
      - name: source-repo
        type: git
    tasks:
      - name: skaffold-unit-tests
        taskSpec:
          inputs:
            resources:
              - name: workspace
                type: git
          steps:
            - name: build-and-push
              image: gcr.io/kaniko-project/executor
              command:
                - /kaniko/executor
              args:
                - --destination=gcr.io/my-project/gohelloworld
        resources:
          inputs:
            - name: workspace
              resource: source-repo

Additional Info

The context for this is running ProwJobs as Tekton Pipelines.
Prow now has a controller that can run ProwJobs that define a PipelineRun spec, but currently the referenced Pipeline and PipelineResources must already exist in the cluster. It would be neat if a ProwJob could specify all the needed information in the PipelineRun spec instead of having to reference other CRDs that must be separately managed by our automation.

[bobcatfish]

Seems totally reasonable to have this! Thanks for opening the issue @cjwagner

The context for this is running ProwJobs as Tekton Pipelines.

Sort of related, I'd also like to bring your attention to #859: we'd really like our Pipeline and Task definitions to live in our repo alongside our code. So another possible way to address your use case could be for Prow to be aware of where the Pipeline and Task definitions are stored in the repo, and apply them before running?

(Would still need some way to stamp out and/or embed PipelineResources tho)

[bobcatfish]

(Would still need some way to stamp out and/or embed PipelineResources tho)

p.s. going to propose pipelineresource embedding for https://github.com/tektoncd/pipeline/issues/544 anyway :D

[dibyom]

@pritidesai is working on this!

[bobcatfish]

/assign pritidesai

[tekton-robot]

@bobcatfish: GitHub didn't allow me to assign the following users: pritidesai.

Note that only tektoncd members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign pritidesai

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bobcatfish]

/assign pritidesai

[bobcatfish]

Design doc for PipelineSpec embedding: https://docs.google.com/document/d/1at7nFsC-4OIILXwAC--EujlF9eJ_yiVt3cBWMRJbIdc/edit#

[bobcatfish]

I'm going to reopen this b/c we don't have Task embedding yet.

[pritidesai]

will start working on Task embedding after PR #1353 and #1333 are merged since these PRs are perquisites for taskSpec under pipelineSpec.

[vdemeester]

@bobcatfish @pritidesai I think all is done now, right ?

[pritidesai]

hey @vdemeester the taskSpec is still pending, just started working on it this week.

[vdemeester]

@pritidesai ok :+1: I thought it was tracked in another issue :stuck_out_tongue_closed_eyes: