Argo-cd: gitops secrets

[gregdurham]

I asked a previous question, in relation to helm plugins, and having more experience with ArgoCD, I determined it made more sense for our environment to utilize helm template to render out k8s manifests and store those in git, rather than using tiller.

The issue now is, how to handle secrets in a safe manner without committing creds in the clear in git.
The solution I am using now is to use sealed secrets, and it works...just not really easy for handling multiple environments and clusters.

I would really like a solution integrated into argocd for git-crypt or SOPS which allows tighter control of secrets, with sops, we can utilize a cloud providers KMS, and tie a user's IAM creds to that.

Is there any feature on the roadmap to support such functionality?

Thanks!

[jessesuen]

Argo CD needs to remain un-opinionated about how secrets are managed in git, simply because there's just so many ways to do it, and there's no one-size-fits-all solution that satisfies everyone's use case, especially when a tight integration with a cloud provider's KMS is desired, or some other central secret store (e.g. S3, Vault). Off the top of my head, here are some ways people are doing secrets with GitOps:

• sealed secrets - https://github.com/bitnami-labs/sealed-secrets
• helm secrets - https://github.com/futuresimple/helm-secrets
• kustomize secret generator plugins - https://groups.google.com/forum/#!msg/kubernetes-sig-cli/MY6hjg2-J-0/bDfwF89SAwAJ

This needed flexibility is precisely why repo-server customization and config management plugins were built.

One solution that I've been considering that might work well with AWS, is:

1. bind an IAM role to the argocd-repo-server pods (e.g. using KIAM)
2. this IAM role should have privileges to either: (a) access the KMS key to decrypt an encrypted secret stored in git, or (b) retrieve the plain-text value of the secret from a source other than git (e.g. s3 bucket, vault)
3. if using kustomize, customize the repo-server with a kustomize secret generator plugin, which would use the mechanism in step 2 to return the secret.

With the above approach, because a single IAM role is bound to the single argocd-repo-server, the tenants of an Argo CD instance would need to be trusted, or at least the kustomize secret generator plugin would need to be designed in such a way that it would only retrieve secrets associated with the repo and not return secrets across git repos.

In any case, it's unlikely that Argo CD will form an opinion on this, at least until clear patterns form on how GitOps and secrets.

[larslevie]

Using the new secret generator plugin requires using a unreleased version and calling kustomize with the --enable_alpha_goplugins_accept_panic_risk flag.

Using a custom version of kustomize is documented but it doesn't look like there is a way to invoke it with the enable_alpha_goplugins_accept_panic_risk flag. Is that correct, or am I not looking in the right place?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alexec]

Re-opening for discussion

[jcooklin]

@alexec We'd like to the sops kustomize secret generator plugin with argocd. Is or will this be supported? Thanks.

[yujunz]

You can always build your own image for argocd-repo-server.

See https://argoproj.github.io/argo-cd/operator-manual/custom_tools/

[alexec]

See https://argoproj.github.io/argo-cd/operator-manual/secret-management/.

[devstein]

@gregdurham @jcooklin

We switched to Argo CD + SOPS to manage our K8s secrets the same way we manage the rest of our K8 resources. To do this, we create a kustomize plugin for SOPS encoded resources and a commit hook to prevent people from committing unencrypted secrets.

https://github.com/viaduct-ai/kustomize-sops#argo-cd-integration

[imranismail]

I built this to use secretsmanager with argocd and kustomize: https://github.com/imranismail/external-secret

[omerlh]

Maybe we can integrate Kamus easily - it has built-in support for encrypting secret for a specific application, so it fit the terms used by argocd. The encryption is done using external KMS, so multiple environments should be no-burner.

(full disclosure: I'm the maintainer of Kamus)

[rchenzheng]

What's the recommended way to fetch Vault secrets without redeploying?

[MattLud]

Is there any guidance on setting up stuff like SOP or helm plugins - helm init happens on each application if I recall correctly.

[devstein]

@MattLud If you are referring to SOPS, we made a plugin, called KSOPS for kustomize and Argo CD.

I don't have any advice on helm plugins 😞

[MattLud]

@devstein - Yeah - I was hoping specifically for guidance on helm plugins since that recommendation is there. Specifically, I'm curious how ${HELM_HOME} should be setup for the plugin directory - should I use a custom workflow to install the plugin every time ala https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/ ?

[devstein]

@MattLud With my limited knowledge of helm plugins, that makes sense to me. You either would inject the plugins via an initContainer and override the HELM_HOME or create a custom docker image with whatever you need.

If you are concerned with the configure of HELM_HOME in the Argo CD image, I recommend kubectl exec into to explore the directory structure or run the docker image itself and override the entrypoint

[MattLud]

Ok - got it helm secrets working. It is not nearly as straightforward as overriding helm secrets as Argo reprovisions a new helm home every run it seems. But I did get it working. Where is the best place for me to dump the doc on doing it?

[nemo83]

@MattLud I'd be interested in knowing how you did that. We use helm secrets with Sops and I would love to integrate it in ArgoCD. Can you create a 1-pager github repo in your account and link here? Much appreciated.

[MattLud]

Hi there - Yes I can put something together on what I did. I don't know how
much more refining it will need, but some direction on this would have been
better than nothing.

On Fri, Mar 6, 2020 at 7:43 AM Giovanni Gargiulo [e-mail]
wrote:

@MattLud https://github.com/MattLud I'd be interested in knowing how
you did that. We use helm secrets with Sops and I would love to integrate
it in ArgoCD. Can you create a 1-pager github repo in your account and link
here? Much appreciated.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/argoproj/argo-cd/issues/1364?email_source=notifications&email_token=AACTA7NRP2Q32LYHRRSO4O3RGD4WNA5CNFSM4HDLUNQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOBMNYA#issuecomment-595773152,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACTA7OSSIGZZF2HIMFWBM3RGD4WNANCNFSM4HDLUNQA
.

[yann-soubeyrand]

@nemo83 here are instructions on how we use Sops with Helm: https://github.com/camptocamp/docker-argocd. We don't use Helm Secrets plugin but I think it should be compatible or at least fairly easy to adapt.

[nhuray]

Just a quick comment about what I did for my company. I took a different approach using a Sops Kubernetes Controller: https://github.com/isindir/sops-secrets-operator/

I'm using Helm to create SopsSecret object (CRD) and the controller use the KMS key (in my case GCP) to decrypt the secret and to create a regular Secret object.

That approach really looks like what's doing Sealed Secret but the encryption/decryption key is outside the Kubernetes cluster (in Cloud KMS) which reduce maintenance in case of DR (recreate the entire cluster).

[xsreality]

Hi there - Yes I can put something together on what I did. I don't know how much more refining it will need, but some direction on this would have been better than nothing.
…
On Fri, Mar 6, 2020 at 7:43 AM Giovanni Gargiulo @.*> wrote: @MattLud https://github.com/MattLud I'd be interested in knowing how you did that. We use helm secrets with Sops and I would love to integrate it in ArgoCD. Can you create a 1-pager github repo in your account and link here? Much appreciated. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1364?email_source=notifications&email_token=AACTA7NRP2Q32LYHRRSO4O3RGD4WNA5CNFSM4HDLUNQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOBMNYA#issuecomment-595773152>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACTA7OSSIGZZF2HIMFWBM3RGD4WNANCNFSM4HDLUNQA .

Curious to know how you integrated helm secrets and sops with argocd. Can you share some info on this please?

[ibmdcb]

@nemo83 here are instructions on how we use Sops with Helm: https://github.com/camptocamp/docker-argocd. We don't use Helm Secrets plugin but I think it should be compatible or at least fairly easy to adapt.

I tried this out and works like a charm. very simple to use and easy to follow. Much appreciated.

[kvaps]

Well. I was trying to implement workaround for git-crypt support, but this is far from ideal:
https://github.com/kvaps/argocd-git-crypt

I would prefer to have native hooks on git clone

[kvaps]

Hi folks, I found better workaround to implement git-crypt for ArgoCD, thus now it is working much better.

I even wrote small article about my advantages:
https://itnext.io/configure-custom-tooling-in-argo-cd-a4948d95626e#567a

However It would be nice to add native support for persistent gpg-keys from the ArgoCD side (see: https://github.com/argoproj/argo-cd/issues/4316)

[arthurk]

@kvaps I'm interested in reading your article but medium wants me to create an account to do so. Is it available on another platform to read?

[kvaps]

@arthurk what's a pity, sure, I copied its content direct on GitHub project
https://github.com/kvaps/argocd-git-crypt#configure-custom-tooling-in-argo-cd

[ittennull]

Hi guys,
I'd like to add another solution to the Argocd's pool of secret management tools.
I created a project that lets Argocd use sops-encrypted secrets in Helm applications.
https://gitlab.com/ittennull/sopshelm
It's easy to install and works great!