Vault: feature request for vault-operator for kubernetes

[raoofm]

I just wanted to request a new feature to have a vault-operator to manage vault ha cluster on kubernetes.

like https://coreos.com/operators/#try

[rezaloo]

@raoofm A Vault operator, with support for HA, is available through the Tectonic Vault Open Cloud service: https://coreos.com/open-cloud-services/#vault

[raoofm]

@rezaloo thanks, yes I'm aware but I opened this issue for a non tectonic kubernetes user and I expect this to be a very common use case and would add much value to vault as well.

[raoofm]

@jefferai Is this on the radar?

[jefferai]

Nope.

[mcwienczek]

+1 for this

[philips]

Introducing Vault Operator Project. Blog: https://coreos.com/blog/introducing-vault-operator-project Github: https://github.com/coreos/vault-operator

[sidewinder12s]

I assume this can probably be closed out with the OSS release of the operator.

[raoofm]

thanks @philips

[raoofm]

@jefferai can this be reopened. vault-operator needs maintainers
https://github.com/coreos/vault-operator/issues/332

It seems like a very common use case inline with OSS kubernetes.

[matyix]

We have a Vault operator which is actively maintained and used in production by us and our customers. It's a very feature rich operator, but should you have any requirements let us know: https://github.com/banzaicloud/bank-vaults.

Besides other features, these are the ones which are missing from the CoreOS operator:

• Automatic Vault initialization
• Root Token and Unseal Keys encrypted and stored in cloud KMS systems (Azure Key Vault, AWS KMS, GCP KMS ,Alibaba KMS)
• Also they can be stored in Kubernetes Secrets (however this is not supposed to be used in production, because the current limitations of Kubernetes Secrets, see this doc for more details)
• Automated unsealing
• Automated re/configuration of Vault based on a YAML/JSON file like: Auth backends, Secret backends, and policies
• It is not tied to etcd at all, supports multiple storage backends (e.g. cloud provider storages)

@raoofm @jefferai @mcwienczek @sidewinder12s

[tamalsaha]

We are working on our own open source Vault operator https://github.com/kubevault

[bonifaido]

Please note that the Banzai Vault Operator is based on the new operator-framework and we have described it in numerous blog posts already:

• https://banzaicloud.com/blog/vault-operator/
• https://banzaicloud.com/blog/vault-unsealing/

[raoofm]

thanks @matyix @tamalsaha @bonifaido
I'll take a look

[sethvargo]

(Personal opinions to follow). I think an operator is an overkill for what's required here. I recommend you take a look at sethvargo/vault-on-gke and kelseyhightower/vault-on-google-kubernetes-engine.

Both those setups give you:

• Vault HA - The Vault cluster is deployed in HA mode backed by Google Cloud Storage
• Production Hardened - Vault is deployed according to the production hardening guide.
• Auto-Init and Unseal - Vault is automatically initialized and unsealed at runtime. The unseal keys are encrypted with Google Cloud KMS and stored in Google Cloud Storage
• Full Isolation - The Vault cluster is provisioned in it's own Kubernetes cluster in a dedicated GCP project that is provisioned dynamically at runtime. Clients connect to Vault using only the load balancer and Vault is treated as a managed external service.

The Vault servers are deployed as StatefulSets, which give you guaranteed (re)start ordering (important for upgrades) and consistent naming.

After talking with some folks on Google's security team, it's also generally a best practice to run Vault in a dedicated namespace or even a dedicated cluster. This negates most of the benefits of an operator, since you'd be accessing Vault simply by an IP address rather than through k8s internals anyway.

Once you're inside a container, you can use vault-kubernetes-authenticator as an initContainer to grab the initial Vault token for you.

In terms of @matyix's list of requirements:

• Automatic Vault initialization
• Automated unsealing
• Root Token and Unseal Keys encrypted and stored in cloud KMS systems (Azure Key Vault, AWS KMS, GCP KMS ,Alibaba KMS)

This is handled by vault-init specifically on GCP. You could fork and adapt it to your cloud provider of choice. Auto-init/unseal is also an enterprise feature you can pay for for added availability. The process for init and initial unseal is very straightforward, and the code is there in OSS.

• Also they can be stored in Kubernetes Secrets (however this is not supposed to be used in production, because the current limitations of Kubernetes Secrets, see this doc for more details)

There's a lot of reasons this isn't feasible at the moment, the biggest of which being Kubernetes secrets have no concept of expiration/renewal and Kubernetes secrets are accessible in plaintext to anyone with admin access to the etcd cluster

• Automated re/configuration of Vault based on a YAML/JSON file like: Auth backends, Secret backends, and policies

This would be outside of what I'd expect even an operator to do. Usually you'd have a central CI/CD server manage this along with capturing your config as code. If anything, this should be a _Vault_ feature, not a function of the operator.

• It is not tied to etcd at all, supports multiple storage backends (e.g. cloud provider storages)

It's generally _not_ recommended that you depend on Kubernetes' etcd for things. The things I linked above leverage Google Cloud Storage, but that could be swapped for Cloud Spanner, EC2, or any other configuration - just update the Vault configuration in the YAML file.

[bonifaido]

I would like to dismiss some concerns and defend the Operator concept a little bit here:

• Basically, an operator (neither the original CoreOS nor the Banzai Cloud version) doesn't block you from running Vault in a separate Kubernetes namespace or cluster, you can run your app in cluster A, run Vault in cluster B and the operator in cluster B or C as you like.
• Kubernetes Secrets: we use this only for development purposes, especially when we are running a whole cluster in minikube, where no cloud features are available locally. As mentioned, not recommended for production use.
• About the API based configuration: in our case, we put a declarative YAML file (no need for scripting in CI/CD jobs) next to the Operator which provisions Vault already, it has all the necessary credentials to do that (no need to distribute those credentials again to another place).
• Init and Unseal are available for AWS, GCP, and Alibaba and Azure (Azure wasn't part of Vault Enterprise when we did this feature) and Alibaba is also requested by our users. This whole feature is set is open-source, no need to reimplement those.
• About etcd: I think no Vault operator used Kubernetes's etcd storage ever for Vault's storage backend, the CoreOS and the Banzai Cloud vault-operator both provision an etcd instance on top of Kubernetes if needed. However we prefer Cloud Provider storage as well.
• If we found something that our users or we missed from Vault we have integrated it into the Operator, if we see a preferable alternative in Vault itself we probably will throw out our separate implementation and rely on Vault as much as possible.

[jefferai]

About etcd: I think no Vault operator used Kubernetes's etcd storage ever for Vault's storage backend, the CoreOS and the Banzai Cloud vault-operator both provision an etcd instance on top of Kubernetes if needed. However we prefer Cloud Provider storage as well.

Just want to comment on this point: it's not just Kubernetes' etcd storage that is the problem, it's all etcd storage. The CoreOS team pledged to support and maintain the etcd storage mechanism, but after the RedHat acquisition that dried up, and currently bug/issue reports for the etcd storage mechanism do not get responses or get addressed. Any vault-operator implementation using etcd is not just unsupported by the Vault team but currently is unsupported by anyone at all.

[raoofm]

/cc @xiang90 @hasbro17

@philips mentioned that etcd-operator will be maintained. https://github.com/coreos/etcd-operator/issues/1719#issuecomment-380943135

[jefferai]

etcd-operator isn't the etcd storage backend for Vault.

[raoofm]

@jefferai 😄 I agree, banzaicloud/bank-vaults use etcd-operator to provision etcd backend, so I mentioned

[catsby]

Hello - I'm going to close this issue for now, thank you for participating in the discussion. HashiCorp has recently released a Helm chart for Vault, you can find more details on that here:

• https://www.vaultproject.io/docs/platform/k8s/index.html

Thanks!