Pipeline: Unable to use secrets with build-bot serviceAccount

[thavlik]

Expected Behavior

A task should use secrets as per the documentation.

Actual Behavior

The TaskRun finishes with this error (as per kubectl get tr build-my-image-task-run -o yaml):

status:
  conditions:
  - lastTransitionTime: "2019-08-02T19:18:58Z"
    message: 'build step "step-build-and-push" is pending with reason "rpc error:
      code = Unknown desc = Error response from daemon: Get https://registry.beebs.dev/v2/thavlik/kaniko-executor/manifests/latest:
      no basic auth credentials"'

Output of kubectl describe pod:

Name:           build-my-image-task-run-pod-5437de
Namespace:      default
Priority:       0
Node:           worker-pool-jdly/10.136.243.56
Start Time:     Fri, 02 Aug 2019 14:16:58 -0500
Labels:         tekton.dev/task=build-my-image-task
                tekton.dev/taskRun=build-my-image-task-run
Annotations:    kubectl.kubernetes.io/last-applied-configuration:
                  {"apiVersion":"tekton.dev/v1alpha1","kind":"Task","metadata":{"annotations":{},"name":"build-my-image-task","namespace":"default"},"spec":{"inp...
                tekton.dev/ready: 
Status:         Pending
IP:             10.244.2.35
Controlled By:  TaskRun/build-my-image-task-run
Init Containers:
  step-credential-initializer-lsc9n:
    Container ID:  docker://caa46d0989acb8c308ffda37a204bba2174c87f466fe20e10a07cfac0470c6e6
    Image:         gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/creds-init@sha256:c0235af1723068e6806def1d998436cde5d93ff1c38a94b9c92410f5f01bcb26
    Image ID:      docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/creds-init@sha256:c0235af1723068e6806def1d998436cde5d93ff1c38a94b9c92410f5f01bcb26
    Port:          <none>
    Host Port:     <none>
    Command:
      /ko-app/creds-init
    Args:
      -ssh-git=ssh-key=github.com
      -docker-config=regcred
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 02 Aug 2019 14:17:01 -0500
      Finished:     Fri, 02 Aug 2019 14:17:02 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      HOME:  /builder/home
    Mounts:
      /builder/home from home (rw)
      /var/build-secrets/regcred from secret-volume-regcred-hkjrt (rw)
      /var/build-secrets/ssh-key from secret-volume-ssh-key-t7rf6 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from build-bot-token-7whm5 (ro)
      /workspace from workspace (rw)
  create-dir-default-image-output-2jrft:
    Container ID:  docker://3e6b162f4a21393db74439bcb4b6ee442f2e77b68de7e27123c489769546d168
    Image:         gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/bash@sha256:157b21c4b29a4f2aa96d52add55781f211cc8101df36657b82089119b2fc4004
    Image ID:      docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/bash@sha256:157b21c4b29a4f2aa96d52add55781f211cc8101df36657b82089119b2fc4004
    Port:          <none>
    Host Port:     <none>
    Command:
      /ko-app/bash
    Args:
      -args
      mkdir -p /builder/home/image-outputs/builtImage
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 02 Aug 2019 14:17:03 -0500
      Finished:     Fri, 02 Aug 2019 14:17:03 -0500
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /builder/home from home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from build-bot-token-7whm5 (ro)
      /workspace from workspace (rw)
  step-place-tools:
    Container ID:  docker://17407a1ef43714421b8a9ee0299fef1ff553ea07d7d8534a7c8f986af49a6102
    Image:         gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint@sha256:a424ab773b89e13e5e03ff90962db98424621b47c1bb543ec270783cfd859faf
    Image ID:      docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint@sha256:a424ab773b89e13e5e03ff90962db98424621b47c1bb543ec270783cfd859faf
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      cp /ko-app/entrypoint /builder/tools/entrypoint
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 02 Aug 2019 14:17:04 -0500
      Finished:     Fri, 02 Aug 2019 14:17:04 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      HOME:  /builder/home
    Mounts:
      /builder/home from home (rw)
      /builder/tools from tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from build-bot-token-7whm5 (ro)
      /workspace from workspace (rw)
Containers:
  step-git-source-my-git-6bvlt:
    Container ID:  docker://47677db885c32a02a41d7165631ab809361308bf5a0fdaf536449a88daf5853f
    Image:         gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init@sha256:2e5217266f515f91be333d5f8abcdc98bb1a7a4de7b339734e10fd7b972eeb5f
    Image ID:      docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init@sha256:2e5217266f515f91be333d5f8abcdc98bb1a7a4de7b339734e10fd7b972eeb5f
    Port:          <none>
    Host Port:     <none>
    Command:
      /builder/tools/entrypoint
    Args:
      -wait_file
      /builder/downward/ready
      -post_file
      /builder/tools/0
      -wait_file_content
      -entrypoint
      /ko-app/git-init
      --
      -url
      https://github.com/thavlik/my-git
      -revision
      master
      -path
      /workspace/docker-source
    State:          Running
      Started:      Fri, 02 Aug 2019 14:17:05 -0500
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:  /builder/home
    Mounts:
      /builder/downward from downward (rw)
      /builder/home from home (rw)
      /builder/tools from tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from build-bot-token-7whm5 (ro)
      /workspace from workspace (rw)
  step-build-and-push:
    Container ID:  
    Image:         registry.beebs.dev/thavlik/kaniko-executor
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /builder/tools/entrypoint
    Args:
      -wait_file
      /builder/tools/0
      -post_file
      /builder/tools/1
      -entrypoint
      /kaniko/executor
      --
      --dockerfile=Dockerfile
      --destination=registry.beebs.dev/thavlik/kaniko-executor
      --context=/workspace/my-git
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:           /builder/home
      DOCKER_CONFIG:  /builder/home/.docker/
    Mounts:
      /builder/home from home (rw)
      /builder/tools from tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from build-bot-token-7whm5 (ro)
      /workspace from workspace (rw)
  step-image-digest-exporter-build-and-push-98lzp:
    Container ID:  docker://b25b407df76bce62f88bf1e353c9e45e1ee1bbac188fc6d9466ad05bfdace402
    Image:         gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/imagedigestexporter@sha256:aae9c44ed56f0d30530a2349f255c4977a6d8d4a497dfdca626b51f35bf229b4
    Image ID:      docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/imagedigestexporter@sha256:aae9c44ed56f0d30530a2349f255c4977a6d8d4a497dfdca626b51f35bf229b4
    Port:          <none>
    Host Port:     <none>
    Command:
      /builder/tools/entrypoint
    Args:
      -wait_file
      /builder/tools/1
      -post_file
      /builder/tools/2
      -entrypoint
      /ko-app/imagedigestexporter
      --
      -images
      [{"name":"my-image","type":"image","url":"registry.beebs.dev/thavlik/kaniko-executor","digest":"","OutputImageDir":"/builder/home/image-outputs/builtImage"}]
      -terminationMessagePath
      /builder/home/image-outputs/termination-log
    State:          Running
      Started:      Fri, 02 Aug 2019 14:17:05 -0500
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:  /builder/home
    Mounts:
      /builder/home from home (rw)
      /builder/tools from tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from build-bot-token-7whm5 (ro)
      /workspace from workspace (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tools:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  downward:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['tekton.dev/ready'] -> ready
  workspace:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  secret-volume-ssh-key-t7rf6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ssh-key
    Optional:    false
  secret-volume-regcred-hkjrt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  regcred
    Optional:    false
  build-bot-token-7whm5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  build-bot-token-7whm5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                       Message
  ----     ------     ----                  ----                       -------
  Normal   Scheduled  20m                   default-scheduler          Successfully assigned default/build-my-image-task-run-pod-5437de to worker-pool-jdly
  Normal   Pulled     20m                   kubelet, worker-pool-jdly  Container image "gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/creds-init@sha256:c0235af1723068e6806def1d998436cde5d93ff1c38a94b9c92410f5f01bcb26" already present on machine
  Normal   Created    20m                   kubelet, worker-pool-jdly  Created container step-credential-initializer-lsc9n
  Normal   Started    20m                   kubelet, worker-pool-jdly  Started container step-credential-initializer-lsc9n
  Normal   Created    20m                   kubelet, worker-pool-jdly  Created container create-dir-default-image-output-2jrft
  Normal   Pulled     20m                   kubelet, worker-pool-jdly  Container image "gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/bash@sha256:157b21c4b29a4f2aa96d52add55781f211cc8101df36657b82089119b2fc4004" already present on machine
  Normal   Started    20m                   kubelet, worker-pool-jdly  Started container create-dir-default-image-output-2jrft
  Normal   Pulled     20m                   kubelet, worker-pool-jdly  Container image "gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint@sha256:a424ab773b89e13e5e03ff90962db98424621b47c1bb543ec270783cfd859faf" already present on machine
  Normal   Created    20m                   kubelet, worker-pool-jdly  Created container step-place-tools
  Normal   Created    20m                   kubelet, worker-pool-jdly  Created container step-git-source-my-git-6bvlt
  Normal   Started    20m                   kubelet, worker-pool-jdly  Started container step-place-tools
  Normal   Pulled     20m                   kubelet, worker-pool-jdly  Container image "gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init@sha256:2e5217266f515f91be333d5f8abcdc98bb1a7a4de7b339734e10fd7b972eeb5f" already present on machine
  Normal   Started    20m                   kubelet, worker-pool-jdly  Started container step-git-source-my-git-6bvlt
  Normal   Pulled     20m                   kubelet, worker-pool-jdly  Container image "gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/imagedigestexporter@sha256:aae9c44ed56f0d30530a2349f255c4977a6d8d4a497dfdca626b51f35bf229b4" already present on machine
  Normal   Created    20m                   kubelet, worker-pool-jdly  Created container step-image-digest-exporter-build-and-push-98lzp
  Normal   Started    20m                   kubelet, worker-pool-jdly  Started container step-image-digest-exporter-build-and-push-98lzp
  Normal   Pulling    20m (x2 over 20m)     kubelet, worker-pool-jdly  Pulling image "registry.beebs.dev/thavlik/kaniko-executor"
  Warning  Failed     20m (x2 over 20m)     kubelet, worker-pool-jdly  Failed to pull image "registry.beebs.dev/thavlik/kaniko-executor": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.beebs.dev/v2/thavlik/kaniko-executor/manifests/latest: no basic auth credentials
  Warning  Failed     20m (x2 over 20m)     kubelet, worker-pool-jdly  Error: ErrImagePull
  Warning  Failed     5m20s (x65 over 20m)  kubelet, worker-pool-jdly  Error: ImagePullBackOff
  Normal   BackOff    19s (x88 over 20m)    kubelet, worker-pool-jdly  Back-off pulling image "registry.beebs.dev/thavlik/kaniko-executor"

Steps to Reproduce the Problem

1. Push a kaniko image to a private repository
2. List a known working k8s dockerconfigjson secret in the secrets section of the ServiceAccount used in the TaskRun
3. Observe ErrImagePull with kubectl describe on the TaskRun-created pod:

Additional Info

I am currently trying to use a kaniko image from a private registry, but I have also used the one from gcr.io and ran into git SSH issues suggesting the container is not utilizing any of the secrets. So the issue seems pertinent to secrets in general.

Here is the manifest for the SSH secret and service account:

---
apiVersion: v1
kind: Secret
metadata:
  name: ssh-key
  annotations:
    tekton.dev/git-0: github.com # Described below
type: kubernetes.io/ssh-auth
data:
  ssh-privatekey: <redacted>
  # This is non-standard, but its use is encouraged to make this more secure.
  #known_hosts:
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-bot
secrets:
  - name: ssh-key
  - name: regcred

If necessary, here is the Task and TaskRun:

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: my-git
spec:
  type: git
  params:
    - name: revision
      value: master
    - name: url
      value: https://github.com/thavlik/my-git
---
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: my-image
spec:
  type: image
  params:
    - name: url
      value: registry.beebs.dev/thavlik/my-image
---
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-my-image-task
spec:
  inputs:
    resources:
      - name: docker-source
        type: git
    params:
      - name: pathToDockerFile
        type: string
        description: The path to the dockerfile to build
        default: /workspace/docker-source/Dockerfile
      - name: pathToContext
        type: string
        description:
          The build context used by Kaniko
          (https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts)
        default: /workspace/docker-source
  outputs:
    resources:
      - name: builtImage
        type: image
  steps:
    - name: build-and-push
      image: registry.beebs.dev/thavlik/kaniko-executor
      # specifying DOCKER_CONFIG is required to allow kaniko to detect docker credential
      env:
        - name: "DOCKER_CONFIG"
          value: "/builder/home/.docker/"
      command:
        - /kaniko/executor
      args:
        - --dockerfile=${inputs.params.pathToDockerFile}
        - --destination=${outputs.resources.builtImage.url}
        - --context=${inputs.params.pathToContext}
---
apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: build-my-image-task-run
spec:
  serviceAccount: build-bot
  taskRef:
    name: build-my-image-task
  inputs:
    resources:
      - name: docker-source
        resourceRef:
          name: my-git
    params:
      - name: pathToDockerFile
        value: Dockerfile
      - name: pathToContext
        value: /workspace/my-git
  outputs:
    resources:
      - name: builtImage
        resourceRef:
          name: my-image    

[dacleyra]

as a guess,
pipeline resource may have to have a url in the form of
[email protected]/thavlik/my-git.git

https://github.com/tektoncd/pipeline/blob/master/examples/taskruns/taskrun-git-ssh.yaml#L40

[thavlik]

Thanks for the tip! I will give this a try and post back!

[rakhbari]

I have the same ErrImagePull issue as @thavlik and mine has nothing to do with Kaniko. It's a fairly straight-forward Task and TaskRun that's attempting to pull a builder image and issue commands to it to build app JAR artifact.

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: tektondemo-source-git
spec:
  type: git
  params:
  - name: revision
    value: master
  - name: url
    value: https://github.corp.ebay.com/cna-working-group/tektoncd-poc
---
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-tektondemo-artifacts
spec:
  inputs:
    resources:
      - name: workspace
        type: git
        targetPath: source
    params:
      - name: raptorio-builder
        description: The path to the raptor-io build image
        default: ecr.vip.ebayc3.com/ciaas/raptorio-builder
  steps:
    - name: mvn-wrapper
      image: ${inputs.params.raptorio-builder}
      command: ["/bin/bash"]
      args: ["-c", "mvn -N io.takari:maven:wrapper -Dmaven=3.5.2"]
    - name: mvn-chmod
      image: ${inputs.params.raptorio-builder}
      command: ["/bin/bash"]
      args: ["-c", "chmod +x /workspace/source/mvnw"]
    - name: mvn-build
      image: ${inputs.params.raptorio-builder}
      command: ['/bin/bash']
      args: ["-c", "cd /workspace/source && ./mvnw clean install -s settings.xml"]
    - name: list-jar
      image: ${inputs.params.raptorio-builder}
      command: ['/bin/bash']
      args: ["-c", "cd /raptor-io-workspace && ls -l"]

      volumeMounts:
        - name: custom-volume
          mountPath: /raptor-io-workspace

  volumes:
    - name: custom-volume
      emptyDir: {}
---
apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: build-tektondemo-artifacts-task-run
spec:
  serviceAccount: build-bot
  taskRef:
    name: build-tektondemo-artifacts
  inputs:
    resources:
      - name: workspace
        resourceRef:
          name: tektondemo-source-git

Here's the build-bot account as described by kubectl:

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2019-08-06T20:05:26Z"
  name: build-bot
  namespace: default
  resourceVersion: "422248"
  selfLink: /api/v1/namespaces/default/serviceaccounts/build-bot
  uid: 7daf64d9-378a-4395-b75d-490ff0ec8b0a
secrets:
- name: github-secret
- name: quay-secret
- name: ecr-secret
- name: build-bot-token-2m88x

I have double and triple-checked the validity of all the secrets listed under secrets. They all absolutely work fine. The builder image in question is in a private internal company registry ecr.vip.ebayc3.com/ciaas/raptorio-builder. The creds that should allow it to be pulled are in ecr-secret as such:

apiVersion: v1
kind: Secret
type: kubernetes.io/basic-auth
metadata:
  name: ecr-secret
  annotations:
    tekton.dev/docker-0: https://ecr.vip.ebayc3.com
stringData:
  username: <redacted>
  password: <redacted>

I can successfully docker login to ecr.vip.ebayc3.com using the above credentials and do docker pull on the same image with no problems.

Yet, soon after the pod runs I get:

  Normal   Pulling            6m37s                 kubelet, tekton-cd  Pulling image "ecr.vip.ebayc3.com/ciaas/raptorio-builder"
  Warning  Failed             6m36s                 kubelet, tekton-cd  Error: ImagePullBackOff
  Warning  Failed             6m36s                 kubelet, tekton-cd  Failed to pull image "ecr.vip.ebayc3.com/ciaas/raptorio-builder": rpc error: code = Unknown desc = failed to resolve image "ecr.vip.ebayc3.com/ciaas/raptorio-builder:latest": no available registry endpoint: failed to fetch anonymous token: unexpected status: 401 Unauthorized
  Warning  Failed             6m36s                 kubelet, tekton-cd  Error: ErrImagePull
  Normal   BackOff            6m36s                 kubelet, tekton-cd  Back-off pulling image "ecr.vip.ebayc3.com/ciaas/raptorio-builder"

Any ideas how I can further diagnose this issue?

[vdemeester]

/kind bug
/kind question

[anxinyf]

@rakhbari Hi, have you solved this problem yet? I have a same problem. When ServiceAccount like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-bot
secrets:
  - name: docker-pass
  - name: git-pass

but cannot push image to destination: UNAUTHORIZED. My secret info is right.

[rakhbari]

@anxinyf No, unfortunately I haven't been able to get this to work. I even added imagePullSecret to my ServiceAccount definition as suggested by the developer docs:
https://github.com/tektoncd/pipeline/tree/master/docs/developers#entrypoint-rewriting-and-step-ordering

But even that didn't help. Frankly I'm not sure if this is an issue with Tekton or an issue with Quay v3. We use RedHat Quay v3 as our private Docker repo and the above developer doc clearly states that you must have an imagePullSecret defined, but like I said, it didn't help.

The only way we've been able to get past this is to make the image public in Quay. It's not a fix, just a workaround for now. I've got to put some time into deep analysis of this issue because we can't go to "production" with a requirement to make images public in Quay.

[anxinyf]

@rakhbari I create a configmap with file ~/.docker/config.json, then mount it to kaniko image, like this:
kubectl create configmap docker-config --from-file=/root/.docker/config.json
And,

...
      env:
      - name: DOCKER_CONFIG
        value: "/builder/home/.docker/"
      volumeMounts:
      - name: docker-config
        mountPath: /builder/home/.docker/
  volumes:
  - name: docker-config
    configMap:
      name: docker-config

serviceAccount like:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-bot
secrets:
  - name: git-pass

it seems to be viable for me.

but I have a question about Secret：

apiVersion: v1
kind: Secret
metadata:
  annotations:
    tekton.dev/git-0: https://github.com
    tekton.dev/git-1: https://gitlab.com
    tekton.dev/docker-0: https://gcr.io
type: kubernetes.io/basic-auth
stringData:
  username: <cleartext non-encoded>
  password: <cleartext non-encoded>

in this Secret, git repo and docker repo have the same username and password?

[vdemeester]

@anxinyf indeed :)
@thavlik is this still an issue ?

[rakhbari]

@anxinyf @vdemeester Sorry for the very late reply. This is no longer an issue for me. I stopped using PipelineResource as of pipelines v1beta1 so now I just use the image: attribute in my Task step directly and that retrieves the builder image directly from our internal Quay registry with no problems.

However, in my service account definition, I had to place my Quay secret under both imagePullSecrets and secrets. Originally I only had it just under imagePullSecrets and that didn't seem to work. I kept getting 401 errors during the TaskRun. But as soon as I added the same Quay secret under secrets, that got rid of that error and the image is able to be pulled with no problems.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.