Helmfile: Feat: Allow simple Vault integration

[AndresPineros]

Currently there are many users that want to integrate Vault with Kubernetes, but there are no high level tools for this. The current Kubernetes AuthMethod for Vault is too complex and coupled to the definition of the charts. This proposal could allow for a Vault integration with Kubernetes that is simple and decoupled.

This could be the structure of a helmfile integrated with Vault:

releases:
  - chart: .
    name: release-test
    version: 0.1.0
    values:
      - application:
           mysql_url: jdbc.....
           mysql_user: localtest
           mysql_password: localtest
    vault:
      active: true
      secrets:
        - vaultPath: secret-engine/my-application
          key: mysql.username
          targetVar: application.mysql_user
        - vaultPath: secret-engine/my-application
          key: mysql.password
          targetVar: application.mysql_password

Basically the project could use the VAULT_ADDR and VAULT_TOKEN environment variables to query all the secrets from Vault using the vaultPath and the key to inject them to the helm release using --set.

So, for that example the --set flags of the helm upgrade -i ... command would look like:

--set application.mysql_user=<queried-secret> --set application.mysql_password=<queried-secret>

If the vault.active field is set to false, or the vault field is not used, the release will just use the "localtest" values.

The main advantage I see for this proposal is that it allows centralization of secrets. Usually storing encrypted secrets in repositories result in secret-management-madness as there is no unique source of truth in case multiple repositories require the same secret. Vault solves this. Of course, this approach only addresses static Vault secrets (secrets in git repos are static too). For dynamic Vault secrets, I think there's no other alternative than the Kubernetes Auth Method.

There is one security issue with this proposal: Tiller has to use the --storage (Secret) backend instead of the ConfigMap backend to avoid fetching injected secrets with helm get values <release-name>. Although, I think this is also a problem for the helm-secrets pluign.

[AndresPineros]

I would really like to PR this, but I don't know Golang. I just started learning it to be able to contribute, but I don't know how long it can take for me to learn and do a decent PR.

[mumoshu]

@AndresPineros Hey!

there is no unique source of truth in case multiple repositories require the same secret

Yeah, I believe this is definitely the thing we should consider.

A possible work-around as of today would be to use templating. That is, include something like {{ exec "./your/cmd/to/fetch/vault/secret/to/dump/yaml/or/json" (list "path.to.key") }} within a values.yaml.gotmpl under secrets: field of your helmfile.yaml.

Would it work?

[sstarcher]

I think this would be a great feature to directly support and essentially change the secrets handling into a plug-able system.

[mumoshu]

And please let me plug-in that my own use-case requires an integration with AWS Secrets Manager :)

[mumoshu]

So I was wondering if it is possible to integrate a tool like gomplate to helmfile.

Relying on gomplate datasources, we can reliably add support for many backends including vault.

An example helmfile.yaml featuring the feature would look like:

releases:
  - chart: .
    name: release-test
    version: 0.1.0
    datasources:
      myvault: vault://vault.example.com:8200///secret-engine/my-application
    values:
      - application:
           mysql_url: jdbc.....
           # Note that {{` and `}} must be required to defer template execution. Otherwise it is executed on loading helmfile.yaml, which doesnt work due to that no datasources are loaded yet
           mysql_url: {{` {{ datasource "myvault" "mysql.username" }} `}}
           mysql_password: {{` {{ datasource "myvault" "mysql.password" }} `}}

[osterman]

Data sources like gomplate would be great. Before I saw this issue, today I was thinking how it would be nice to access SSM Parameter store directly.

[mumoshu]

An alternative idea would be to focus on making secrets consumption from any backend declarative, while making the changes required to helmfile minimum.

That is, allow defining any secrets backend within a helmfile.yaml:

secretsProvider:
- name: vault
  command: "bin/helmfile-secrets-provider-vault"
  protocols:
  - "vault"

releases:
-  chart: .
    name: release-test
    version: 0.1.0
    secrets:
      - vault://vault.example.com:8200////secret-engine/my-application/?application.mysql_user=mysql.user,application.mysql_password=mysql.password

In the above example, bin/helmfile-vault-provider is executed like bin/helmfile-vault-provider 'vault://vault.example.com:8200////secret-engine/my-application/?...' > $temp_file_created_by_helmfile to produce a values.yaml-like file containing the secret .

The produced secrets file should be subject to caching proposed in #444

[sstarcher]

I would think you would want to move more of the configuration for the backend out of the secrets block so you don't have to repeat it.

[mumoshu]

Thanks!

That sounds good. Then my best idea would be https://github.com/roboll/helmfile/issues/392#issuecomment-455061273.

Reimplementing gomplate datasources or integrating gomplate, along with my proposed extension to helmfile's template engine and helmfile.yaml would make the repetition minimum, while allowing flexibility to either source a single secret value or the whole YAML data.

For my point of writing any secrets provider within a helmfile.yaml, adding new gomteplate datasource(or its reimplementation)s like exec would work.

[mumoshu]

I now think we can start with something simple like the below:

templates:
  vault: &vault
    exec: "vault read secrets/foo/bar/{{.key}}"
  vault: &awssecret
    exec: "aws secretmanager get ...."

releases:
- name: myapp
  chart: ./mychart
  secrets:
  - secrets.yaml #helm-secrets
  - <<: *vault
     key: myvaultsecret
  - <<: *awssecret
     key: myawssecret.key

[mumoshu]

And the above should work fine with #444

[sstarcher]

That works, but it is going to get very ugly very fast.

[red8888]

@mumoshu godaddy has something pretty cool for hydrating k8s secrets from AWS Secrets Manager: https://github.com/godaddy/kubernetes-external-secrets

BUT you gotta get AWS creds to access Secrets Manager in there to begin with somehow. It would be great if helmfile could retrieve those AWS creds from secrets manager and create k8s secrets for the external secrets manager service to use

[mumoshu]

@red8888 Hey! Yeah I've even considered to add integration with my aws-secret-operator but I'm still guesting what the real benefit It provides.

For your case, are you talking about making Helmfile able to import SecretsManager secrets as Helm chart values, or something more advanced?

AWS creds from secrets manager and create k8s secrets for the external secrets manager service to use

This has nothing to do with kubernetes-external-secrets. Probably you meant to reimplement part of kubernetes-external-secrets into Helmfile so that it creates K8s secrets from SecretsManager secrets?

[red8888]

@mumoshu yeah thats what i was thinking. I was not aware of https://github.com/mumoshu/aws-secret-operator

kubernetes-external-secrets does mostly the same thing looks like.

I just meant a service used for hydrating k8s secrets from aws secrets manager needs AWS creds to access it. using kubernetes-external-secrets I have been manually scripting that part of the deployment (pulling the AWS creds from Secrets Manger and creating a k8s secret for kubernetes-external-secrets to use). After that point kubernetes-external-secrets dynamically creates k8s secrets for me from secrets manager (through the custom resources like your service does)

its not a big deal to script it myself, but would be neat if I could have helm pull the initial creds. that might be asking to much though

[mumoshu]

@red8888 Thanks for clarifying! I think I understood your problem now.

Your use-case should be addressed with #745 which has AWS Secrets Manager as one of supported backends.

You can combine it with the incubator/raw Helm chart so that you can create whatever K8s resources from helmfile values.

[etiennejournet]

I now think we can start with something simple like the below:

templates:
  vault: &vault
    exec: "vault read secrets/foo/bar/{{.key}}"
  vault: &awssecret
    exec: "aws secretmanager get ...."

releases:
- name: myapp
  chart: ./mychart
  secrets:
  - secrets.yaml #helm-secrets
  - <<: *vault
     key: myvaultsecret
  - <<: *awssecret
     key: myawssecret.key

I haven't been able to understand or use this example, could you clarify ?
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors: line 12: field exec not found in type state.TemplateSpec line 24: cannot unmarshal !!map into string

EDIT ==> went further but still can't get it to work :

templates:                                                                      
  vault: &vault                                                                 
    exec: "vault kv get -field={{.key}} secrets/test"                       

releases:                                                                       
- name: chart                                              
  chart: chart                             
  namespace: chart                                                                                                     
  secrets:                                                                      
  - <<: *vault                                                                  
    key: root_password

Goes :

executing "stringTemplate" at <.key>: can't evaluate field key in type state.EnvironmentTemplateData

[mumoshu]

@etiennejournet Hey! The feature that supports the example isn't implemented yet.
Fortunately, we now have a better alternative #906 that you can use.