Pipeline: Able to start pipelinerun from pipelinerun spec without providing value for workspace and the pipelinerun stays is `Running(Started)` state

Created on 11 Aug 2020  ·  8Comments  ·  Source: tektoncd/pipeline

Expected Behavior

Pipelinerun should fail with some validation error

Actual Behavior

Pipelinerun starts and stays in Running(Started) state

Steps to Reproduce the Problem

  1. Create a task with workspace
  2. Create a pipeline with workspace using the above task
  3. Create pipelinerun spec file to start the pipelinerun for above pipeline and don't provide any value for the workspace
  4. start the pipelinerun using kubectl apply -f

Additional Info

  • Kubernetes version:

    Output of kubectl version:

    Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0+b3b92b2", GitCommit:"b3b92b2", GitTreeState:"clean", BuildDate:"2020-07-15T09:27:21Z", GoVersion:"go1.14.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.3+012b3ec", GitCommit:"012b3ec", GitTreeState:"clean", BuildDate:"2020-07-24T07:23:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

  • Tekton Pipeline version:

    Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

    $ tkn version
Client version: 0.11.0
Pipeline version: v0.14.3
Triggers version: v0.6.1

Command logs

[varadhya@localhost workspace-testing]$ cat workspace-pipelinerun.yaml 
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: test-workspace-pipeline-run-7
spec:
  params:
  - name: first-param
    value: simple value
  pipelineRef:
    name: test-workspace-pipeline
  serviceAccountName: pipeline
  timeout: 1h0m0s
  workspaces:
  - name: shared-workspace
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ oc apply -f workspace-pipelinerun.yaml 
pipelinerun.tekton.dev/test-workspace-pipeline-run-7 created
[varadhya@localhost workspace-testing]$ tkn pr ls --limit 1
NAME                            STARTED          DURATION   STATUS
test-workspace-pipeline-run-7   36 seconds ago   ---        Running(Started)
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ tkn pr describe test-workspace-pipeline-run-7
Name:              test-workspace-pipeline-run-7
Namespace:         veeresh-testing
Pipeline Ref:      test-workspace-pipeline
Service Account:   pipeline
Timeout:           1h0m0s
Labels:
 tekton.dev/pipeline=test-workspace-pipeline

🌡️  Status

STARTED          DURATION   STATUS
49 seconds ago   ---        Running(Started)

📦 Resources

 No resources

⚓ Params

 NAME            VALUE
 ∙ first-param   simple value

🗂  Taskruns

 No taskruns
[varadhya@localhost workspace-testing]$

Spec files

[varadhya@localhost workspace-testing]$ cat workspace-task.yml 
apiVersion: tekton.dev/v1beta1 
kind: ClusterTask 
metadata:
  name: test-workspace-clustetask
spec:
  steps:
  - name: write-message
    image: ubuntu
    script: |
      #!/usr/bin/env bash
      set -xe
      echo hello! > $(workspaces.messages.path)/message
      cat $(workspaces.messages.path)/message
  workspaces:
  - name: messages
    description: The folder where we write the message to
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ cat workspace-pipeline.yaml 
apiVersion: tekton.dev/v1beta1 
kind: Pipeline 
metadata:
  name: test-workspace-pipeline
spec:
  params:
    - name: first-param
      description: this is the first param for Pipeline
  workspaces:
    - name: shared-workspace
      description: this is a shared workspace
  tasks:
    - name: first-task
      taskRef:
        name: test-workspace-clustetask
        kind: ClusterTask
      workspaces:
        - name: messages
          workspace: shared-workspace
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ cat workspace-pipelinerun.yaml 
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: test-workspace-pipeline-run-7
spec:
  params:
  - name: first-param
    value: simple value
  pipelineRef:
    name: test-workspace-pipeline
  serviceAccountName: pipeline
  timeout: 1h0m0s
  workspaces:
  - name: shared-workspace
[varadhya@localhost workspace-testing]$
kinbug lifecyclrotten
Source
picture VeereshAradhya

All 8 comments

Hey @VeereshAradhya we recently added a feature where you can specify a default that is controller wide (https://github.com/tektoncd/pipeline/blob/master/docs/workspaces.md#using-workspaces-in-tasks re default-task-run-workspace-binding), so maybe that default is being used?

(I do wonder @sbwsg @jerop if it would make sense to make it more obvious when the default is being used, e.g. require "defualt" is provided for a workspace vs just allowing it)

@VeereshAradhya you might get more info about what's going on if you look at the pod, e.g. kubectl get pod -l tekton.dev/PipelineRun=test-workspace-pipeline-run-7

bobcatfish picture bobcatfish on 11 Aug 2020

Hm. This is odd. I definitely see a problem in the syntax of the PipelineRun:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: test-workspace-pipeline-run-7
spec:
  params:
  - name: first-param
    value: simple value
  pipelineRef:
    name: test-workspace-pipeline
  serviceAccountName: pipeline
  timeout: 1h0m0s
  workspaces:
  - name: shared-workspace

Here the PipelineRun is "binding" a Workspace with name shared-workspace but isn't actually providing the volume configuration to go with it. We would normally expect something more along the lines of this:

workspaces:
- name: shared-workspace
  persistentVolumeClaim: # ... pvc info ...

Our code does check the number of volume configurations in the binding so I am a bit surprised this isn't erroring out here.

Agree with @bobcatfish that it would be useful to see the pod that was created here. It would also be useful to know the contents of the default configmap (kubectl get configmap -n tekton-pipelines config-defaults).

(I do wonder @sbwsg @jerop if it would make sense to make it more obvious when the default is being used, e.g. require "defualt" is provided for a workspace vs just allowing it)

I'd like to discuss this a bit more before forming an opinion on it. It seems, at least initially to me, to be a bit counter to the purpose of the feature to require it be opt-in at the Task/TaskRun level.

Sneaky edit to add: the Optional Workspaces TEP calls out that any Workspace marked optional will _not_ receive the default taskrun workspace. That's mentioned in this section here.

sbwsg picture sbwsg on 11 Aug 2020

@sbwsg @bobcatfish I think I have not written the proper heading for the issue. The issue is I am able to create a pipelinerun (with status Running(Started)) without providing volume configuration for the workspace in pipelinerun spec. The pipelinerun does not create any taskruns/pods. When I do kubectl apply -f to a pipelinerun spec which does not provide volume configuration for workspace, it should throw an error instead of creating pipelinerun.
I have reproduced the issue in new namespace, below are the command execution logs and pipeline controller logs, let me know if anything more needs to be added

[varadhya@localhost workspace-testing]$ kubectl get pods
No resources found.
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ tkn t ls
No Tasks found
[varadhya@localhost workspace-testing]$ tkn p ls
No Pipelines found
[varadhya@localhost workspace-testing]$ cat workspace-task.yml 
apiVersion: tekton.dev/v1beta1 
kind: Task 
metadata:
  name: test-workspace-task
spec:
  steps:
  - name: write-message
    image: ubuntu
    script: |
      #!/usr/bin/env bash
      set -xe
      echo hello! > $(workspaces.messages.path)/message
      cat $(workspaces.messages.path)/message
  workspaces:
  - name: messages
    description: The folder where we write the message to
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ cat workspace-pipeline.yaml 
apiVersion: tekton.dev/v1beta1 
kind: Pipeline 
metadata:
  name: test-workspace-pipeline
spec:
  params:
    - name: first-param
      description: this is the first param for Pipeline
  workspaces:
    - name: shared-workspace
      description: this is a shared workspace
  tasks:
    - name: first-task
      taskRef:
        name: test-workspace-task
        kind: Task
      workspaces:
        - name: messages
          workspace: shared-workspace
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ kubectl apply -f workspace-task.yml 
task.tekton.dev/test-workspace-task created
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ kubectl apply -f workspace-pipeline.yaml 
pipeline.tekton.dev/test-workspace-pipeline created
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ tkn t ls
NAME                  DESCRIPTION   AGE
test-workspace-task                 14 seconds ago
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ tkn p ls
NAME                      AGE             LAST RUN   STARTED   DURATION   STATUS
test-workspace-pipeline   6 seconds ago   ---        ---       ---        ---
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ cat workspace-pipelinerun.yaml 
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: test-workspace-pipeline-run-7
spec:
  params:
  - name: first-param
    value: simple value
  pipelineRef:
    name: test-workspace-pipeline
  serviceAccountName: pipeline
  timeout: 1h0m0s
  workspaces:
  - name: shared-workspace
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ kubectl apply -f workspace-pipelinerun.yaml 
pipelinerun.tekton.dev/test-workspace-pipeline-run-7 created
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ tkn pr ls
NAME                            STARTED         DURATION   STATUS
test-workspace-pipeline-run-7   5 seconds ago   ---        Running(Started)
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ tkn tr ls
No TaskRuns found
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ 
[varadhya@localhost workspace-testing]$ kubectl get pods
No resources found.

[varadhya@localhost workspace-testing]$ kubectl logs tekton-pipelines-controller-6889f6f9c7-649nk -n tekton-pipelines | tail -n4
{"level":"error","logger":"tekton.github.com-tektoncd-pipeline-pkg-reconciler-pipelinerun.Reconciler","caller":"controller/controller.go:480","msg":"Reconcile error","commit":"6d12d1e","error":"1 error occurred:\n\t* error creating TaskRun called test-workspace-pipeline-run-7-first-task-6g8cr for PipelineTask first-task from PipelineRun test-workspace-pipeline-run-7: admission webhook \"validation.webhook.pipeline.tekton.dev\" denied the request: validation failed: expected exactly one, got neither: workspace.configmap, workspace.emptydir, workspace.persistentvolumeclaim, workspace.secret, workspace.volumeclaimtemplate\n\n","stacktrace":"github.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller/controller.go:480\ngithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller/controller.go:466\ngithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller.(*Impl).RunContext.func3\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller/controller.go:403"}
{"level":"info","logger":"tekton.github.com-tektoncd-pipeline-pkg-reconciler-pipelinerun.Reconciler","caller":"controller/controller.go:467","msg":"Reconcile failed. Time taken: 29.753684ms","commit":"6d12d1e","knative.dev/traceid":"9caa610c-ac67-452b-a382-8a2fdf928918","knative.dev/key":"veeresh-pipelinerun-issue/test-workspace-pipeline-run-7"}
{"level":"info","logger":"tekton.event-broadcaster","caller":"record/event.go:281","msg":"Event(v1.ObjectReference{Kind:\"PipelineRun\", Namespace:\"veeresh-pipelinerun-issue\", Name:\"test-workspace-pipeline-run-7\", UID:\"29fd788b-e740-4a2d-9927-dd2f58797889\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"331447\", FieldPath:\"\"}): type: 'Warning' reason: 'TaskRunCreationFailed' Failed to create TaskRun \"test-workspace-pipeline-run-7-first-task-6g8cr\": admission webhook \"validation.webhook.pipeline.tekton.dev\" denied the request: validation failed: expected exactly one, got neither: workspace.configmap, workspace.emptydir, workspace.persistentvolumeclaim, workspace.secret, workspace.volumeclaimtemplate","commit":"6d12d1e"}
{"level":"info","logger":"tekton.event-broadcaster","caller":"record/event.go:281","msg":"Event(v1.ObjectReference{Kind:\"PipelineRun\", Namespace:\"veeresh-pipelinerun-issue\", Name:\"test-workspace-pipeline-run-7\", UID:\"29fd788b-e740-4a2d-9927-dd2f58797889\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"331447\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* error creating TaskRun called test-workspace-pipeline-run-7-first-task-6g8cr for PipelineTask first-task from PipelineRun test-workspace-pipeline-run-7: admission webhook \"validation.webhook.pipeline.tekton.dev\" denied the request: validation failed: expected exactly one, got neither: workspace.configmap, workspace.emptydir, workspace.persistentvolumeclaim, workspace.secret, workspace.volumeclaimtemplate\n\n","commit":"6d12d1e"}
[varadhya@localhost workspace-testing]$
VeereshAradhya picture VeereshAradhya on 12 Aug 2020

Got it, thanks for clarifying - that'll teach me to rush through reading the original issue >.<

This does indeed appear to be a bug in the validation of the PipelineRun. I'll work on reproducing my side and adding a fix.

sbwsg picture sbwsg on 12 Aug 2020
👍1

It appears that there are two discrete bugs here. The first is that we're not validating the PipelineRun workspaces. The second is that somehow the PipelineRun reconciler is submitting an invalid TaskRun spec but then not putting the PipelineRun into a failed state when the TaskRun fails due to workspace validation error.

A fix for the PipelineRun validation is here: https://github.com/tektoncd/pipeline/pull/3096

sbwsg picture sbwsg on 12 Aug 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

tekton-robot picture tekton-robot on 10 Nov 2020

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

tekton-robot picture tekton-robot on 10 Dec 2020

@sbwsg @VeereshAradhya should we close this one ?

vdemeester picture vdemeester on 10 Dec 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

castlemilk picture castlemilk  ·  4Comments
chmouel picture chmouel  ·  3Comments
sujithjoseph picture sujithjoseph  ·  3Comments
hrishin picture hrishin  ·  3Comments
bobcatfish picture bobcatfish  ·  4Comments