Pipeline: PipelineRun hangs in Pending state

[castlemilk]

Expected Behavior

Pipeline should run singular task, as it would in pre v0.5.0

Actual Behavior

Pipeline stun in Pending state, waiting for a PVC to be provisioned which hasn't been created by the controller.

Logs will be observed warning:

{"level":"info","logger":"controller.taskrun-controller","caller":"taskrun/taskrun.go:334","msg":"Successfully reconciled taskrun demo-pipeline-run-1-build-skaffold-web-lvgrc/default with status: &apis.Condition{Type:\"Succeeded\", Status:\"Unknown\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0xbf42083cb35e1ed1, ext:4499218940429, loc:(*time.Location)(0x27a75a0)}}}, Reason:\"Pending\", Message:\"pod status \\\"PodScheduled\\\":\\\"False\\\"; message: \\\"persistentvolumeclaim \\\\\\\"demo-pipeline-run-1-pvc\\\\\\\" not found\\\"\"}","knative.dev/controller":"taskrun-controller"}

and

{"level":"warn","logger":"controller.pipeline-controller","caller":"artifacts/artifacts_storage.go:149","msg":"the configmap has no data","knative.dev/controller":"pipeline-controller"}

Steps to Reproduce the Problem

deploy the following pipeline:

---
# This demo modifies the cluster (deploys to it) you must use a service
# account with permission to admin the cluster (or make your default user an admin
# of the `default` namespace with default-cluster-admin.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-cluster-admin
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: skaffold-image-leeroy-web
spec:
  type: image
  params:
  - name: url
    value: gcr.io/christiewilson-catfactory/leeroy-web
---
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: skaffold-git
spec:
  type: git
  params:
  - name: revision
    value: v0.32.0
  - name: url
    value: https://github.com/GoogleContainerTools/skaffold
---
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-push
spec:
  inputs:
    resources:
    - name: workspace
      type: git
    params:
    - name: pathToDockerFile
      description: The path to the dockerfile to build
      default: /workspace/workspace/Dockerfile
    - name: pathToContext
      description: The build context used by Kaniko (https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts)
      default: /workspace/workspace
  outputs:
    resources:
    - name: builtImage
      type: image
  steps:
  - name: build-and-push
    image: gcr.io/kaniko-project/executor:v0.9.0
    # specifying DOCKER_CONFIG is required to allow kaniko to detect docker credential
    env:
    - name: "DOCKER_CONFIG"
      value: "/builder/home/.docker/"
    command:
    - /kaniko/executor
    args:
    - --dockerfile=${inputs.params.pathToDockerFile}
    - --destination=${outputs.resources.builtImage.url}
    - --context=${inputs.params.pathToContext}
---
apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
  name: demo-pipeline
spec:
  resources:
  - name: source-repo
    type: git
  - name: web-image
    type: image
  tasks:
  - name: build-skaffold-web
    taskRef:
      name: build-push
    params:
    - name: pathToDockerFile
      value: Dockerfile
    - name: pathToContext
      value: /workspace/workspace/examples/microservices/leeroy-web
    resources:
      inputs:
      - name: workspace
        resource: source-repo
      outputs:
      - name: builtImage
        resource: web-image
---
apiVersion: tekton.dev/v1alpha1
kind: PipelineRun
metadata:
  name: demo-pipeline-run-1
spec:
  pipelineRef:
    name: demo-pipeline
  serviceAccount: 'default'
  resources:
  - name: source-repo
    resourceRef:
      name: skaffold-git
  - name: web-image
    resourceRef:
      name: skaffold-image-leeroy-web

The Pod for running the pipeline will be created, but without a PVC, consequently the Pod is stuck in a Pending state, waiting for a referenced PVC, which hasn't been created by the tekton-controller.

The full pipeline defined here then works, which contains multiple tasks in one pipeline

Tekton Version - v0.5.0
Kubernetes Versio - 1.13

[bobcatfish]

Oh noes, I'm seeing the same thing :(((( Not sure how this slipped by our tests but I guess we'll find out!!

    startTime: 2019-07-11T21:46:50Z
    taskRuns:
      demo-pipeline-run-1-build-skaffold-web-8nvhj:
        pipelineTaskName: build-skaffold-web
        status:
          conditions:
          - lastTransitionTime: 2019-07-11T21:46:51Z
            message: 'pod status "PodScheduled":"False"; message: "persistentvolumeclaim
              \"demo-pipeline-run-1-pvc\" not found"'
            reason: Pending
            status: Unknown
            type: Succeeded
          podName: demo-pipeline-run-1-build-skaffold-web-8nvhj-pod-1f1a43
          startTime: 2019-07-11T21:46:50Z

I wanted to make a new 0.5.1 release with https://github.com/tektoncd/pipeline/pull/1061 but we should fix this first!

I'm guessing this is a bug with https://github.com/tektoncd/pipeline/pull/1007 ~so (icky) way to work around this would be to link the resources 😓~ nvm, there's only one Task, that's probably the problem like you said @castlemilk - and why the CI didnt catch it 😩

[castlemilk]

Awesome! yea I found if you create a second step that links or depends on the first it'll created a PVC and enable the pipeline to run.

[bobcatfish]

Sorry again about this @castlemilk and thanks for the quick report! I've created https://github.com/tektoncd/pipeline/tree/release-v0.5.1 with our 2 bug fixes and I'll make 0.5.1 release first thing monday EST :) 🙏

[bobcatfish]

Okay https://github.com/tektoncd/pipeline/releases/tag/v0.5.1 should have this fixed! (It's back to making an extra PVC unfortunately, but we'll have a better fix for that via #937 hopefully!)

Thanks again for the report and your patience @castlemilk ! ❤️