Pipeline: v0.13.0: unsuccessful cred copy: ".ssh" from "/tekton/creds" to "/root"

[bitsofinfo]

Expected Behavior

I should be able to opt-out of the auto seeding of tekton creds into each step and just mount my own secrets via workspaces under /root. Let the user have complete control and disable some of the automatic stuff tekton does around this. For steps which use images that rely on specific things in the user's home dir (in this case /root) I need to craft this directory in a specific way and this tekton cred stuff gets in the way.

Actual Behavior

Task goes into Error state after 1st step. Each step in my task that does this ends up w/ this in logs. NOTE the serviceAccount this is running as has NO bound secrets.

2020/06/09 13:47:33 unsuccessful cred copy: ".ssh" from "/tekton/creds" to "/root": unable to open destination: open /root/.ssh/known_hosts: read-only file system

Overall in the tekton dashboard, the task just appears to be in a hung state.

Steps to Reproduce the Problem

Note I have "disable-home-env-overwrite":"true","disable-working-directory-overwrite":"true"

1. Mount a secret backed workspace to a task at /root/.ssh
2. Start a pipelinerun for that task

My trigger workspaces:

        workspaces:
          - name: ssh-secrets
            secret:
              secretName: my-ssh-config 
              items:
                - key: ssh-privatekey
                  path: id_rsa
                  mode: 256
                - key: known_hosts
                  path: known_hosts
                - key: ssh-config
                  path: config

The task def workspaces:

    - name: ssh-secrets
      description: The workspace where we can find ssh secrets
      mountPath: /root/.ssh
      readOnly: true

Additional Info

The serviceAccount that the pipelinerun uses, has no secrets attatched to it.

• Kubernetes version:

1.16.9

• Tekton Pipeline version: 0.13.0

[sbwsg]

this tekton cred stuff gets in the way.

Best, most succinct, description of creds-init I've read yet. Thanks for the issue report!

[sbwsg]

In terms of the "hang" though - could you post the sanitized Pod YAML? The errors you have in the issue report "shouldn't" cause hangs so I'd like to try reproducing to see if I can figure out what's going wrong.

[sbwsg]

/kind bug

[bitsofinfo]

here is a dump of the pod yaml as it sits

apiVersion: v1
kind: Pod
metadata:
  annotations:
    pipeline.tekton.dev/release: devel
    tekton.dev/ready: READY
  creationTimestamp: "2020-06-09T13:47:27Z"
  labels:
    app.kubernetes.io/managed-by: tekton-pipelines
    tekton.dev/pipeline: deploy-v1
    tekton.dev/pipelineRun: deploy-v1-mycat-wjc7x
    tekton.dev/pipelineTask: deploy-v1
    tekton.dev/task: deploy-v1
    tekton.dev/taskRun: deploy-v1-mycat-wjc7x-deploy-v1-lc4pr
    triggers.tekton.dev/eventlistener: myap-listener
    triggers.tekton.dev/trigger: myap-deploy-v1-interceptor
    triggers.tekton.dev/triggers-eventid: chq7n
  name: deploy-v1-mycat-wjc7x-deploy-v1-lc4pr-pod-vcdvl
  namespace: tekton-pipelines
  ownerReferences:
  - apiVersion: tekton.dev/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: TaskRun
    name: deploy-v1-mycat-wjc7x-deploy-v1-lc4pr
    uid: 31267e89-3ea6-4a27-afb4-6030ca06f715
  resourceVersion: "17465340"
  selfLink: /api/v1/namespaces/tekton-pipelines/pods/deploy-v1-mycat-wjc7x-deploy-v1-lc4pr-pod-vcdvl
  uid: fef1e7fe-05c5-4d99-b5d6-af18ef3dc349
spec:
  containers:
  - args:
    - -wait_file
    - /tekton/downward/ready
    - -wait_file_content
    - -post_file
    - /tekton/tools/0
    - -termination_path
    - /tekton/termination
    - -entrypoint
    - /bin/bash
    - --
    - -c
    - |

      ls -al ~
      cd ~
      pwd
      whoami

      cat ~/.gitconfig

      ls -al ~/.ssh

      ls -al ~/.kube

      APP_NAME=mycat
      APP_VERSION=2.1.3-1
      APP_ENV=stage
      APP_CONTEXT=stage-e1
      TARGET_CLUSTER=stage2
      K8S_NS=myapps
      K8S_TILLER_NS=myapps
      TARGET_CONDUIT=nv
      INVOKER_UID=burt

      PIPELINE_RUN_LINK="https://host1/#/namespaces/tekton-pipelines/pipelineruns/deploy-v1-mycat-wjc7x"

    command:
    - /tekton/tools/entrypoint
    image: bitsofinfo/alpine-bash-curl-jq-yq:1.0.0
    imagePullPolicy: IfNotPresent
    lifecycle:
      postStart:
        exec:
          command:
          - /bin/sh
          - -c
          - cp /workspace/secrets/git /root/.gitconfig
    name: step-setup
    resources:
      requests:
        cpu: "0"
        ephemeral-storage: "0"
        memory: "0"
    terminationMessagePath: /tekton/termination
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /workspace/secrets/git
      name: ws-pflvw
      readOnly: true
    - mountPath: /root/.aws-secrets.yaml
      name: ws-5q7vb
      readOnly: true
    - mountPath: /root/.kube
      name: ws-4sdrx
      readOnly: true
    - mountPath: /root/.ssh
      name: ws-6bsl2
      readOnly: true
    - mountPath: /tekton/tools
      name: tekton-internal-tools
    - mountPath: /tekton/downward
      name: tekton-internal-downward
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/creds
      name: tekton-creds-init-home
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: myap-deployer-token-h6d7c
      readOnly: true
  - args:
    - -wait_file
    - /tekton/tools/0
    - -post_file
    - /tekton/tools/1
    - -termination_path
    - /tekton/termination
    - -entrypoint
    - /entrypoint.sh
    - --
    - --log-level=DEBUG

    command:
    - /tekton/tools/entrypoint
    env:
    - name: USER
      value: tekton-pipeline
    image: privatereg.mystuff.com/lp/my-utils-installer:2.2.13
    imagePullPolicy: IfNotPresent
    lifecycle:
      postStart:
        exec:
          command:
          - /bin/sh
          - -c
          - cp /workspace/secrets/git /root/.gitconfig
    name: step-deploy
    resources:
      requests:
        cpu: "0"
        ephemeral-storage: "0"
        memory: "0"
    terminationMessagePath: /tekton/termination
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /workspace/secrets/git
      name: ws-pflvw
      readOnly: true
    - mountPath: /root/.aws-secrets.yaml
      name: ws-5q7vb
      readOnly: true
    - mountPath: /root/.kube
      name: ws-4sdrx
      readOnly: true
    - mountPath: /root/.ssh
      name: ws-6bsl2
      readOnly: true
    - mountPath: /tekton/tools
      name: tekton-internal-tools
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/creds
      name: tekton-creds-init-home
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: myap-deployer-token-h6d7c
      readOnly: true
  - args:
    - -wait_file
    - /tekton/tools/1
    - -post_file
    - /tekton/tools/2
    - -termination_path
    - /tekton/termination
    - -entrypoint
    - /bin/bash
    - --
    - -c
    - |
      APP_NAME=mycat
      APP_VERSION=2.1.3-1
      APP_ENV=stage
      APP_CONTEXT=stage-e1
      TARGET_CLUSTER=stage2
      K8S_NS=myapps
      K8S_TILLER_NS=myapps
      TARGET_CONDUIT=nv
      INVOKER_UID=burt

    command:
    - /tekton/tools/entrypoint
    image: bitsofinfo/alpine-bash-curl-jq-yq:1.0.0
    imagePullPolicy: IfNotPresent
    name: step-notify
    resources:
      requests:
        cpu: "0"
        ephemeral-storage: "0"
        memory: "0"
    terminationMessagePath: /tekton/termination
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /workspace/secrets/git
      name: ws-pflvw
      readOnly: true
    - mountPath: /root/.aws-secrets.yaml
      name: ws-5q7vb
      readOnly: true
    - mountPath: /root/.kube
      name: ws-4sdrx
      readOnly: true
    - mountPath: /root/.ssh
      name: ws-6bsl2
      readOnly: true
    - mountPath: /tekton/tools
      name: tekton-internal-tools
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/creds
      name: tekton-creds-init-home
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: myap-deployer-token-h6d7c
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: privatereg-mystuff
  initContainers:
  - args:
    - -docker-config=privatereg-mystuff
    command:
    - /ko-app/creds-init
    env:
    - name: HOME
      value: /tekton/creds
    image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/creds-init:v0.13.0@sha256:5206d6880896935ffa53d2d8326a2a0b49be902a04bb92f235b22958473e83d7
    imagePullPolicy: IfNotPresent
    name: credential-initializer
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/creds
      name: tekton-creds-init-home
    - mountPath: /tekton/creds-secrets/privatereg-mystuff
      name: tekton-internal-secret-volume-privatereg-mystuff-zmxpz
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: myap-deployer-token-h6d7c
      readOnly: true
  - command:
    - cp
    - /ko-app/entrypoint
    - /tekton/tools/entrypoint
    image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint:v0.13.0@sha256:0cbfbc4f4ed9cbf6060e12c12d36599b2d8ca3f13e3fd5432adf2c2f9001913d
    imagePullPolicy: IfNotPresent
    name: place-tools
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tekton/tools
      name: tekton-internal-tools
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: myap-deployer-token-h6d7c
      readOnly: true
  nodeName: clstr1-27390584-0
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: myap-deployer
  serviceAccountName: myap-deployer
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: tekton-internal-workspace
  - emptyDir: {}
    name: tekton-internal-home
  - emptyDir: {}
    name: tekton-internal-results
  - emptyDir:
      medium: Memory
    name: tekton-creds-init-home
  - name: tekton-internal-secret-volume-privatereg-mystuff-zmxpz
    secret:
      defaultMode: 420
      secretName: privatereg-mystuff
  - emptyDir: {}
    name: tekton-internal-tools
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations['tekton.dev/ready']
        path: ready
    name: tekton-internal-downward
  - name: ws-pflvw
    secret:
      defaultMode: 420
      items:
      - key: git-config
        path: .gitconfig
      secretName: myapps-cicd-mystuff-git-support
  - name: ws-5q7vb
    secret:
      defaultMode: 420
      items:
      - key: aws-secrets.yaml
        mode: 256
        path: aws-secrets.yaml
      secretName: myapps-cicd-mystuff-deployer-support
  - name: ws-4sdrx
    secret:
      defaultMode: 420
      items:
      - key: config
        mode: 256
        path: config
      secretName: myapps-cicd-mystuff-kubeconfig
  - name: ws-6bsl2
    secret:
      defaultMode: 420
      items:
      - key: ssh-privatekey
        mode: 256
        path: id_rsa
      - key: known_hosts
        path: known_hosts
      - key: ssh-config
        path: config
      secretName: myapps-cicd-mystuff-ssh-config
  - name: myap-deployer-token-h6d7c
    secret:
      defaultMode: 420
      secretName: myap-deployer-token-h6d7c
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-06-09T13:47:33Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-06-09T13:47:27Z"
    message: 'containers with unready status: [step-setup step-deploy]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-06-09T13:47:27Z"
    message: 'containers with unready status: [step-setup step-deploy]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-06-09T13:47:27Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://bb363abe39296bc41a4d786bb889e42aa8f9ef837c395430af187cce8af0097a
    image: privatereg.mystuff.com/lp/my-utils-installer:2.2.13
    imageID: docker-pullable://privatereg.mystuff.com/lp/my-utils-installer@sha256:82cbe503b781a45983851c0f506124dde4099580d1b68028940c8e8df3a7d629
    lastState: {}
    name: step-deploy
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: docker://bb363abe39296bc41a4d786bb889e42aa8f9ef837c395430af187cce8af0097a
        exitCode: 2
        finishedAt: "2020-06-09T13:47:34Z"
        reason: Error
        startedAt: "2020-06-09T13:47:34Z"
  - containerID: docker://8d5be6b5dc3f0406a45a50133ee8860c4fa01c20442b13a9cc8afd8f56ec34b2
    image: bitsofinfo/alpine-bash-curl-jq-yq:1.0.0
    imageID: docker-pullable://bitsofinfo/alpine-bash-curl-jq-yq@sha256:0d32a576efca666234043a7cd652e90d4f53b3e8ac4ffdf1e0367eb2f81d5085
    lastState: {}
    name: step-notify
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2020-06-09T13:47:35Z"
  - containerID: docker://6d0712afc3b2dca124e84dbf89bc00d147c19d6f487201e3c4884c1b57a8533b
    image: bitsofinfo/alpine-bash-curl-jq-yq:1.0.0
    imageID: docker-pullable://bitsofinfo/alpine-bash-curl-jq-yq@sha256:0d32a576efca666234043a7cd652e90d4f53b3e8ac4ffdf1e0367eb2f81d5085
    lastState: {}
    name: step-setup
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: docker://6d0712afc3b2dca124e84dbf89bc00d147c19d6f487201e3c4884c1b57a8533b
        exitCode: 2
        finishedAt: "2020-06-09T13:47:33Z"
        reason: Error
        startedAt: "2020-06-09T13:47:33Z"
  hostIP: 10.218.9.4
  initContainerStatuses:
  - containerID: docker://2f8d987fe5d822176ff5505912c6a092f72e718c10a68721df0e8374ce55f5b6
    image: sha256:b87b9c158e1cec9af0cad96ceda27c8f81cfb24446c42d6fa281f23284dfdc5e
    imageID: docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/creds-init@sha256:5206d6880896935ffa53d2d8326a2a0b49be902a04bb92f235b22958473e83d7
    lastState: {}
    name: credential-initializer
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://2f8d987fe5d822176ff5505912c6a092f72e718c10a68721df0e8374ce55f5b6
        exitCode: 0
        finishedAt: "2020-06-09T13:47:29Z"
        reason: Completed
        startedAt: "2020-06-09T13:47:29Z"
  - containerID: docker://291e4ee6347370424902abefc2935a060e9c182dfa4f5aa53624e64806a7e2fd
    image: sha256:27adf3dd377a234289bc7bf60e8fb9d7463abe7ccaf360fe402ec31b404078b8
    imageID: docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint@sha256:0cbfbc4f4ed9cbf6060e12c12d36599b2d8ca3f13e3fd5432adf2c2f9001913d
    lastState: {}
    name: place-tools
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://291e4ee6347370424902abefc2935a060e9c182dfa4f5aa53624e64806a7e2fd
        exitCode: 0
        finishedAt: "2020-06-09T13:47:30Z"
        reason: Completed
        startedAt: "2020-06-09T13:47:30Z"
  phase: Running
  podIP: 192.168.3.90
  podIPs:
  - ip: 192.168.3.90
  qosClass: BestEffort
  startTime: "2020-06-09T13:47:27Z"

[sbwsg]

There are two separate issues here:

1. First, a feature request to optionally remove creds-init behaviour. @bitsofinfo do you think you'd prefer this kind of control per-Task/TaskRun? Or is it something you'd like to configure at the cluster level through a configmap? Or something else?
2. The creds init code will write empty credential files if no creds are provided. So for example, ~/.ssh/known_hosts will be written even if there are no ssh git creds on the service account attached to the TaskRun. Those empty files are then copied from /tekton/creds in to $HOME/.ssh. In this case /root/.ssh is read-only because it's a mounted Secret so that copy fails. We should only really be creating credential files if there are correctly annotated secrets mounted so I've made #2940 to fix that issue.

[bitsofinfo]

Both top level feature-flag and per Task level configuration. Preference for per Task

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bitsofinfo]

/remove-lifecycle rotten
/reopen

[tekton-robot]

@bitsofinfo: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Shahard2]

any solutions to this one so far?
I'm also facing it..