Argo: Support to get secrets from Vault

[ramanNarasimhan77]

Summary

Is it possible to inject secrets from a secure vault like Hashicorp Vault instead of getting them from a Kubernetes secret?

Eg:

SCENARIO 1 :

I am currently defining input git artifact used to clone the repo and copy to a PV as shown below. Here I am using k8s secret to get the git username and password

name: get-source-code
    inputs:
      artifacts:
      - name: argo-source
        path: /repoData
        git:
          repo: "{{workflow.parameters.git-repo-url}}"
          revision: "{{workflow.parameters.git-revision}}"
          usernameSecret:
            name: "{{workflow.parameters.git-secret-name}}"
            key: username
          passwordSecret:
            name: "{{workflow.parameters.git-secret-name}}"
            key: password
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["whoami; ls -ld /workspace; mkdir /workspace/modelcode; cp -r /repoData/* /workspace/modelcode; echo '\n\nlisting workspace\n';  ls /workspace/modelcode"]
      volumeMounts:
      - name: workspace
        mountPath: /workspace

SCENARIO 2: Docker config for Kaniko builds

The current configuration that I am using is shown below:

Define volume docker-config that reads docker config json from k8s secret docker-registry

volumes:
  - name: docker-config
    secret:
      secretName: docker-registry     # name of an existing k8s secret
      items:
          - key: .dockerconfigjson
            path: config.json

kaniko build step that uses docker-config volume

- name: build-and-push
    inputs:
      parameters:
        - name: model_id
        - name: model_version
        - name: project_name
        - name: current_time
    container:
      image: gcr.io/kaniko-project/executor:debug
      args: ["--dockerfile","Dockerfile",
             "--context","dir://workspace/modelcode",
              "--destination","{{workflow.parameters.docker-repo-url}}/{{inputs.parameters.project_name}}/{{inputs.parameters.model_id}}:{{inputs.parameters.model_version}}"]
      volumeMounts:
        - name: docker-config
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace

Motivation

To improve application security, we are working on moving all our credentials to Hashicorp Vault. Vault supports injecting secrets either through an init container / sidecar / using Container Storage Interface (CSI) plugin

References:

1. https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/
2. https://learn.hashicorp.com/vault/kubernetes/secret-store-driver

[yokiworks]

I am trying to do now exactly the same and looking for a way to load secrets from vault injector. the problem is that I cannot bypass the argo-secret

[alexec]

The core team does not use Vault, so we can't really advise on this. Have you asked in Slack?

[ramanNarasimhan77]

@yokiworks , @alexec

I figured out the kaniko part, as kaniko just expects the docker config to be present at /kaniko/.docker/config.json
I inserted the docker secret into vault and I am able to inject the secret to /kaniko/.docker/config.json by following the tutorial.

For the git input artifact since a kubernetes secret is expected as input, I am not sure how to bypass it. Can the git credentials be read from a mounted volume?

[ramanNarasimhan77]

The core team does not use Vault, so we can't really advise on this. Have you asked in Slack?

Thanks for the pointer, I will post in slack.

[blkperl]

@yokiworks @ramanNarasimhan77 my team is using https://github.com/godaddy/kubernetes-external-secrets for the use case you are describing.

Blog post: https://www.godaddy.com/engineering/2019/04/16/kubernetes-external-secrets/

[yokiworks]

@yokiworks @ramanNarasimhan77 my team is using https://github.com/godaddy/kubernetes-external-secrets for the use case you are describing.

Blog post: https://www.godaddy.com/engineering/2019/04/16/kubernetes-external-secrets/

Thanks! This is how we solve it also for now... but it is less secure I guess than having the secret injected into the argo pod bypassing the k8s secret

[ramanNarasimhan77]

@yokiworks
Hello,
here is how I have ported my workflow to use Vault for now:

High level vault setup steps

Based on https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/

1. Install consul helm chart
2. Install vault-k8s helm chart in ha mode (uses consul backend)
3. Unseal the vault
4. Enable kubernetes auth, set kubernetes auth config
5. Enable versioning for secrets [Allows you to store multiple versions of same secret]
6. Create a vault policy allowing read access to secret/. If you have multiple applications, you could create policy per application and give read access to secret/app-name/
7. Create a vault role that binds the policy with the serviceaccount and namespace that argo workflows use. The service account that is bound to the role should have access to Token Review api (see https://www.vaultproject.io/docs/auth/kubernetes#configuring-kubernetes)

For git step

We cannot use argo's git input artifact as in the example since this needs us to supply k8s secret. Using vault we can mount files to the pod filesystem. So to use https to clone, we could enable git creds store and mount the credentials to ~/git-credentials

For this purpose configure a secret in vault say secret/gitcreds with key as gitcreds and save the value as https://username:[email protected]

Below we are mounting appuser's gitcredentials to /home/appuser/.git-credentials so that git clone can read the credentials from this file
Enable git credential store to automatically supply credentials to git clone from the credential file
git config --global credential.helper store

 - name: get-source-code-https
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-.git-credentials: "secret/gitcreds"
        vault.hashicorp.com/secret-volume-path: "/home/appuser"
        vault.hashicorp.com/role: "app-user"
        vault.hashicorp.com/agent-inject-template-.git-credentials: |
          {{- with secret "secret/gitcreds" -}}
            {{ .Data.data.gitcreds }}
          {{- end }}

Similarly if we want to clone using ssh, we could put the ssh key into vault and mount to ~/.ssh/id_rsa

For the kaniko step

Kaniko requires the docker config to be present at /kaniko/.docker/config.json
I created a secret in vault secret/dockerconfig with dockerconfig as key and json value

Following annotations inject the secret at /kaniko/.docker/config.json

metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-config.json: "secret/dockerconfig"
        vault.hashicorp.com/secret-volume-path: "/kaniko/.docker"
        vault.hashicorp.com/role: "app-user"
        vault.hashicorp.com/agent-inject-template-config.json: |
          {{- with secret "secret/dockerconfig" -}}
            {{ .Data.data.dockerconfig }}
          {{- end }}

[yokiworks]

Thank you @ramanNarasimhan77 this is really neat!
Can we somehow do the same for config.oidc secrets and bypass the need to use a k8s secret for them as well?