Pipeline: Parameters is not supported in csi Volumes

[Yannig]

Expected Behavior

Inject parameters into csi volume definition.

Actual Behavior

If I provide a parameters in volume -> csi, the parameters are not substitued correctly and Kube is complaining about it.

Steps to Reproduce the Problem

1. Use a Task with volumes csi (using Azure vault secrets for example)
2. Run it using a TaskRun

Here an example of Task declaration doing a simple git checkout:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: git-checkout
spec:
  stepTemplate:
    image: alpine/git:v2.24.3
    command: ["git"]
    volumeMounts:
      - name: "secrets"
        mountPath: "/mnt/secrets-store"
        readOnly: true
  params:
    - name: "git-url"
    - name: "secret-provider"
    - name: "store-cred"
  steps:
    - name: "git-config"
      command: ["sh", "-c", "git config --global credential.helper store"]
    - name: "set-credential"
      command:
        - "sh"
        - "-c"
        - |-
            user=`cat /mnt/secrets-store/git-username`
            pass=`cat /mnt/secrets-store/git-password`
            url=`cat /mnt/secrets-store/url`
            echo "https://$user:$pass@$url" > ~/.git-credentials
    - name: "git-checkout"
      args: ["clone", "$(params.git-url)"]
      workingDir: "/workspace"for
  volumes:
    - name: "secrets"
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "$(params.secret-provider)"
        nodePublishSecretRef:
          name: "$(params.store-cred)"

To run this Task, I can use the following TaskRun object (with kubectl create):

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  generateName: git-checkout-
spec:
  taskRef:
    name: git-checkout
  params:
    - name: git-url
      value: https://github.com/my/favorite-repo.git
    - name: secret-provider
      value: azure-kvname
    - name: store-cred
      value: secrets-store-creds

This volume will also need two others objects:

• one with azure credentials:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: secrets-store-creds
data:
  clientid: XXXXX...XXXX
  clientsecret: XXX...XXXX

• another one with Azure vault object definition

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  parameters:
    keyvaultName: "my-vault-test"
    tenantId: "xxx-xxxxx-xxxxxx-xxxxx"
    objects: |
      array:
        - "{ objectName: url, objectType: secret}"
        - "{ objectName: git-username, objectType: secret}"
        - "{ objectName: git-password, objectType: secret}"
    resourceGroup: "my-group"

Additional Info

• Kubernetes version: 1.16.8

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-21T01:25:41Z", GoVersion:"go1.13.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.8-eks-e16311", GitCommit:"e163110a04dcb2f39c3325af96d019b4925419eb", GitTreeState:"clean", BuildDate:"2020-03-27T22:37:12Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version: v0.12.1

[vdemeester]

@Yannig thanks for the issue. Indeed it's not supported yet. We may need to rework our "variable interpolation" implementation as, for now, it only supports a small set of things and is not "extendable" (aka on types we refer and that we don't "know in advance").

/kind feature

[Yannig]

@vdemeester I made a fix for my specific case but maybe I could try to work on something more generic.
What do you think?

[vdemeester]

Right, we need to discuss the more "generic" approach. But we can fix the specific case in the mean time :angel:
/cc @sbwsg @jlpettersson @bobcatfish @skaegi

[jlpettersson]

Is there any specific reason to mount the volume directly here, instead of using a PersistentVolumeClaim as shown in Container Storage Interface (CSI) for Kubernetes GA?

I think the intention with PVC was to decouple the app from storage-specific things (that may be specific to the platform.). In this case Tekton is "your app".

I also understand that all the _in-tree_ storage integrations will in future be more _external_ from the Kubernetes project. Kubernetes podcast about CSI.

In case a PVC is used, this can be used today, as how I understand, by specifying the storageClassName that is declared for your Storage type.

If we are going to have params for all kind of in-tree-volumes, there is a ton of them Kubernetes Volumes

[jlpettersson]

@Yannig this use case come from Azure Secret Store examples, right? they have a few examples with _inline-volume_, but they also have a few examples using PV and PVC - that is methods to decouple the app, e.g. Tekton from the storage implementations. Would any of those satisfy your needs?

[Yannig]

@jlpettersson Here I'm trying to mount a secret from Azure Vault using CSI for this purpose. The following example is close to what I'm trying to do: https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/examples/nginx-pod-secrets-store-inline-volume-secretproviderclass-podid.yaml

[jlpettersson]

@Yannig that would be, like this without _inlining_ the volume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-secrets-store
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass:
        secretProviderClass: "azure-kvname"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-secrets-store
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi
  volumeName: pv-secrets-store
  storageClassName: ""

and then used by a TaskRun or PipelineRun as a PVC workspace by declaring the PVC as:

    persistentVolumeClaim:
      claimName: pvc-secrets-store

by following pv example, pvc example and store pvc example

should work, right?

without adding anyting that is tightly coupled with CSI-volumes to Tekton.

[Yannig]

@jlpettersson Thanks, I got the idea now. I'll do some test and give you feedback then :)

[jlpettersson]

and then used by a TaskRun or PipelineRun as a PVC workspace by declaring the PVC as:

Actually, if you want, this can be set as a PersistentVolumeClaim in the Task, and pass the PVC ClaimName as a Parameter and do this reference (very similar to your initial example), too bad it is not well documented.

  volumes:
    - name: "secrets"
      persistentVolumeClaim:
        name: "$(params.claimName)"

By doing it like this, you can pass claimName as a parameter and do not need to use a PVC workspace in the PipelineRun.

[Yannig]

Hi @jlpettersson

I was able to do some experiment and I get an error while trying to mount this kind of volume:

Events:
  Type     Reason       Age              From                                                  Message
  ----     ------       ----             ----                                                  -------
  Normal   Scheduled    8s               default-scheduler                                     Successfully assigned default/read-az-vault-secret-74bcb4c98b-gqr78 to ip-192-168-20-54.eu-west-1.compute.internal
  Warning  FailedMount  1s (x5 over 8s)  kubelet, ip-192-168-20-54.eu-west-1.compute.internal  MountVolume.NewMounter initialization failed for volume "pv-secrets-store" : volume mode "Persistent" not supported by driver secrets-store.csi.k8s.io (only supports ["Ephemeral"])

The driver does not support to be used through a PV: you must use this kind of volume through a Ephemeral definition (ie directly in a pod definition).

[jlpettersson]

@Yannig sorry for late response on this.

The driver does not support to be used through a PV: you must use this kind of volume through a Ephemeral definition (ie directly in a pod definition).

Ah, I see. This seem to be a newer limitation in recent Kubernetes versions, ref CSI ephemeral inline volumes

In Tekton v0.13, a new feature, TaskRunSpecs was introduced - that allow you to declare _inline volumes_ in the PipelineRun for a specific TaskRun. As elaborated about in https://github.com/tektoncd/pipeline/issues/2680 - this way to declare volumes, let you design Tasks so that they are independent on what kind of volumes they are used with, e.g. a user could declare a PersistentVolumeClaim or a CSI inline volume in the TaskRunSpec - but they use the same Task.

An example, possible from Tekton v0.13+

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: git-clone-ssh-
spec:
  pipelineRef:
    name: pipeline-with-git-clone
  taskRunSpecs:
  - pipelineTaskName: git-clone         # name of task in the Pipeline
    taskPodTemplate:
      volumes:
      - name: secrets
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "azure-kvname"
          nodePublishSecretRef:
            name: "secrets-store-creds"

[Yannig]

Thanks for your answer. I will try to do some experiment and give you feedback as soon as possible.