Vault: Kubernetes Secret Backend

[bartlettc22]

Feature Request:

Would like to see a Kubernetes Secret Backend, similar to the AWS one, in which we could use Vault to provision short-lived Kubernetes credentials for users deployments, etc.

[sudermanjr]

I second this need. Would greatly assist our implementation of Kubernetes security.

[kr3cj]

I third this need. It would solve a big problem that we are dealing with at work. And possibly world hunger. At least the former.

[msvechla]

Also looking for something like this. There is no solution out there for such a scenario. Would be a huge plus!

[mruepp]

We have the same issue...

[jgiles]

TL,DR: I'm planning a Vault plugin backend to do issue Kubernetes bearer tokens for webhook authentication. If you have feedback on the outline below, please let me know.

We also need a solution for this at Paxos. We had planned just to use the PKI backend to issue client certs and use those for authentication, but Kubernetes Dashboard does not support login with client certificates (a blocker for us). You also can't currently write a Kubernetes credential plugin to fetch client certs.

So, we're going to try writing a Vault plugin backend implementing the Kubernetes webhook token authentication protocol:

• An authenticated Vault user can obtain a Kubernetes bearer token from Vault, based on policies allowing access to a Kubernetes "role" defined in Vault.
• The Kubernetes API server will be configured to post its TokenReview authentication webhooks to an unauthenticated "review" endpoint on Vault, which will deny bad/expired tokens access, and for good tokens will assert user membership in the set of Kubernetes groups configured on the "role".

Right now I'm planning to generate the username/UID of the authenticated response on:

• Entity/Alias information from the identity backend, if present
• Vault request DisplayName/ClientToken(hashed), as a fallback

while group the group memberships and "extra" fields are baked into the "role" definition.

This username/uid/group strategy above is based on the presumption that you are managing Kubernetes permissions via groups, and that the username/UID information is most useful for attributing activity to individual people. In that world, it's very helpful to simplify correlation between the Kubernetes audit logs and the Vault audit logs. Defining the Kubernetes group memberships in "roles" but getting username/uid from login information allows Vault operators to give groups of people the same Vault policy (access to the token-issue endpoint for a particular role), mapping to the same Kubernetes permissions, but with uniquely-attributable and hopefully human-comprehensible Kubernetes activity logs.

If you have feedback on the above model, please let me know - We're hoping to have something up in the next week or so.

[jefferai]

Cool!

A few comments

The Kubernetes API server will be configured to post its TokenReview authentication webhooks to an unauthenticated "review" endpoint on Vault, which will deny bad/expired tokens access, and for good tokens will assert user membership in the set of Kubernetes groups configured on the "role".

Is it not possible to give the Kubernetes API a token with which to make the call? (Is it horribly insecure?)

Vault request DisplayName/ClientToken(hashed), as a fallback

I'd avoid DisplayName -- it's entirely possible you could end up with the same display name for very different entities. Hashed req.ClientToken should be sufficient.

[jgiles]

Thank you for the feedback @jefferai !

Is it not possible to give the Kubernetes API a token with which to make the call? (Is it horribly insecure?)

The Kubernetes webhook configuration is specified in kubeconfig format (IMHO, a dubious decision for re-use...). This means that the auth options for Kubernetes requests to Vault are the same as those from Kubernetes clients to the Kubernetes API server. For this case, the options are:

• TLS client certificates (useable with Vault, but only based on server-wide TLS listener configuration)
• An "Authorization: Bearer <token>" header, which does not line up with the X-Vault-Token header expected by Vault

Either way, I think the the requests have to be "unauthenticated" in the Vault sense (they will not be authorized by the presentation of a Vault token and application of Vault policy rules). I'm not incredibly worried about this, because the endpoint is just used for verification. If an attacker has a valid token they could get group information, but they could get the same information (and do much more) by talking to the Kubernetes apiserver. If they do not, spamming to search for valid tokens could likewise be done through the kubernetes apiserver.

If the Authorization header will get piped through to the backend in Request.Headers, then the backend could do its own secondary authentication by issuing and validating tokens for the Kubernetes cluster. Would header get through? The docs on the header field just talk about audit backends.

I'd avoid DisplayName -- it's entirely possible you could end up with the same display name for very different entities. Hashed req.ClientToken should be sufficient.

Thank you for the heads-up! The username/uid determination logic is the bit of this proposal I am least certain about.

[jefferai]

It wouldn't be the first time that we have an unauthenticated endpoint that is only used for verification, but it's of course nicer if Kubernetes can send along a token (which can be scoped for just access to that one endpoint) as this allows better control via ACL/Sentinel policies and so on.

You can control what headers go through to what backends via mount-tuning so Authorization should be able to get sent back there. Alternately, I'm actually thinking about whether we can accept an Authorization-formatter header containing a valid Vault token as the bearer token as an alternative to X-Vault-Token. It might be useful for integration cases like this.

[jgiles]

Updates/Clarifications from related #4562:

For https://github.com/hashicorp/vault/issues/3839#issuecomment-390979103, the Kubernetes cluster will have to "phone home" to Vault with a webhook in order to verify the issued bearer tokens. It's not every request (TokenReview results are cached for a configurable period of time, default 2 minutes). The Kubernetes "webhook authentication" mode basically just delegates token issuance and authentication to some external system. https://github.com/hashicorp/vault/issues/3839#issuecomment-390979103 is about making Vault able to serve as that system.

For our use-case, this is fine - we already have many reasons Kubernetes needs to talk to Vault (e.g. Kubernetes auth for pods, pods getting secrets, underlying EC2 nodes getting secrets).

A potential "feature request" for the proposed backend would be support for client-cert based creds in addition to bearer tokens. Essentially this would involve running a stripped-down version of the PKI backend, with the API tuned for ease-of-use specifically with Kubernetes.

But, this isn't in-scope for the initial implementation... we're putting together the webhook-based authentication largely to get around the Kubernetes-side shortcomings of the client certs.

[jgiles]

@jefferai I was hoping to use req.EntityID to hand out Kubernetes user IDs that were stable over time for a particular Vault user. However, it appears the entity ID is stripped unless the request is going to the sys backend:
https://github.com/hashicorp/vault/blob/95978c208f0b4c0966f3ef2edc1c8ca62cfe3e2e/vault/router.go#L440

I presume that is a security behavior you don't want to get rid of?

If so, could a per-mount-salted EntityID be set on requests so that backends can identify users across logins? (As I understand it, this is how ClientToken is passed to backends).

ClientToken is what I will use for now. I'm guessing to be safe I should salt + hash it again before exposing it to callers?

[jefferai]

Whether you'd want entity id or client token depends on what you're doing with it, which comes down to what your define as a "Vault User" in your perceived workflow.

[jgiles]

I have a draft implementation of the plugin backend up here: https://github.com/paxos-bankchain/vault-plugin-secrets-kubernetes/pull/1. It's working in end-to-end integration tests with a Kubernetes cluster (see the Travis CI build).

The main TODOs are some additional testing (particularly around role management) and documentation.

@jefferai if you have any bandwidth to take a look I would very much appreciate any feedback you have. I've marked the PR up with comments where I'm not sure about stuff, or where I thought some discussion might be warranted.

Anyone else with some Vault familiarity is of course also welcome to chime in.

[jgiles]

Hey @jefferai (or other Vault maintainers), is there anyone who could take a look at https://github.com/paxos-bankchain/vault-plugin-secrets-kubernetes/pull/1, particularly the 4 questions left as comments in the PR? There are still documentation and testing improvements to make, but I'd like to make sure things are heading in the right direction.

We're also interested in upstreaming the plugin for wider distribution - what does that process look like?

[jefferai]

@jgiles I don't see questions?

[jgiles]

They're inline comments on the code - I've made the question more explicit and @mentioned you on each:

• https://github.com/paxos-bankchain/vault-plugin-secrets-kubernetes/pull/1#discussion_r191456636
• https://github.com/paxos-bankchain/vault-plugin-secrets-kubernetes/pull/1#discussion_r191467337
• https://github.com/paxos-bankchain/vault-plugin-secrets-kubernetes/pull/1#discussion_r191471182
• https://github.com/paxos-bankchain/vault-plugin-secrets-kubernetes/pull/1#discussion_r191474310

[jefferai]

Oh, I was looking below the last comments I sent.

[rvadim]

Hi all, there are kubernetes secrets plugin, tested and used in JetBrains infrastructure https://github.com/jetbrains-infra/vault-plugin-secrets-kubernetes . Feel free to create issues and Pull Requests :smiley:

[seanmalloy]

@rvadim are you considering upstreaming your plugin to be included as part of Vault? Or are you looking to just keep this as a separate Jetbrains maintained plugin?

[rvadim]

@seanmalloy we have plan for upstreaming it to Vault, but in any way, we use it and will maintain it.