Vault: Proposal: JOSE Backend with RFC

[ekristen]

This is a proposal of a JOSE backend implementation RFC. It is documented in an HTTP API format that is in keeping with existing secret backend implementation.

Please feel free to comment on the implementation documentation, I'll make updates to the HTTP API in this issue based on feedback. Hopefully once consensus is gathered I'll be able to write the backend for inclusion into the project.

Thank you ahead of time.

backend: jose

Below is the proposed vault jose backend setup.

POST /jose/config/lease

Configures the default lease information for all roles

• lease
• lease_max

POST /jose/config/urls

• base_url

Basically like PKI and defining CRL url, if we define the base URL of vault here, then for things like JWS we can auto-populate the JKU value with a URL back to the JWK format of the public key used.

Example: http://localhost:8200/v1/jose/ - When JWS or JWE are generated with jku set to true, the value will be populated with http://localhost:8200/v1/jose/keys/:name/public

POST /jose/keys/:name

Creates a key that can be used with a role

Parameters

• alg (used for JWS and JWE)
  
  
  
  • need to determine viable algorithms (probably can just use go-jose library)
  
  
• enc (used for JWE)
  
  
  
  • need to determine viable algorithms (probably can just use go-jose library)
  
  
• name (needs to be unique per key)
• private_key (can be provided or generated automatically based on alg type)
• enc_private_key (can be provided or generated automaticallyed based on enc type)
  

  GET /jose/keys/:name

Gets details about a key, access should be restricted using acls

GET /jose/keys/:name/public

Returns the public key as a JWK format, this isn't available for HS algorithm

GET /jose/keys/:name/public/raw

Returns the public key in its original format, this isn't available for HS algorithm

POST /jose/roles/:name

Creates a new role from an already created key

• name (the name of the role)
• type (jwe, jws, jwt)
• key (the name of the key previously created)
• allow_override_at_issue (array of claims that can be overridden at issue time)
  
  
  
  • (pertains to JWT only) the importance of this is that there are times that the subject (aka the user) is going to be different
    
    for each issue of a token, the same for audience if you are targetting different applications, adding these
    
    into this array allows them to be set at issuance
  
  
• allow_custom_payload_at_issue [Note: name could use some work]
  
  
  
  • (pertains to JWT only) there are times that unique information, like some basic user information that is going to change
    
    every issue that would make managing a role for every single user impossible and a large burden on
    
    the vault server
  
  

type: jwt - claims

All claims are optional except JTI, JTI should always be added.

• iss (Issuer) Claim - (this is set at role creation time)
• sub (Subject) Claim - (this is set at role creation time)
• aud (Audience) Claim - (this is set at role creation time)
• exp (Expiration Time) Claim (this is based on the lease) (boolean) default: true - based on lease time
• nbf (Not Before) Claim (this is based on the current time) - (boolean) default: true
  
  
  
  • Note:: this _might_ be worth allowing a user to set at issue time, in the case where you want to create a token ahead of time that will be valid in a week, however this might be out of scope and initial implementation most likely should follow the rule that iat has below.
  
  
• iat (Issued At) Claim (this is based on the current time) - (boolean) default: true
  
  
  
  • Note: this should not be set by the user. To ensure integrity of the tokens, if iat is true, it'll be set using the time on the vault server.
  
  
• jti (JWT ID) Claim - (this is a randomly generated UUID) (required)

type: jwe

• alg (required) - inherited from the key type when defining the role
• enc (optional) - inherited from the key type when defining the role
• zip (optional) - boolean flag to enable zip compression of the JWE
• jku (optional) - boolean flag to include a link to the JWK
• jwk (optional) - boolean flag to include the JWK of the public key for the private key used to secure the token
• kid (optional) - boolean flag to include the key id
• x5u (optional) - boolean flag to include a link to the x509 certificate chain (eg /jose/keys/:name/public/raw)
• x5c (optional) - boolean flag to include a link to include the x509 certificate chain in the token
• x5t (optional) - boolean flag to include the sha1 fingerprint
• x5t#256 (optional) - boolean flag to include the sha256 fingerprint
• typ (optional) - included from the role definition, possibly support setting custom types in the future?
• cty (optional) - allow setting of a content-type

• crit (optional)

  type: jws

• alg (required) - inherited from the key type when defining the rolect

• jku (optional) - boolean flag to include a link to the JWK
• jwk (optional) - boolean flag to include the JWK of the public key for the private key used to secure the token
• kid (optional) - boolean flag to include the key id
• x5u (optional) - boolean flag to include a link to the x509 certificate chain (eg /jose/keys/:name/public/raw)
• x5c (optional) - boolean flag to include a link to include the x509 certificate chain in the token
• x5t (optional) - boolean flag to include the sha1 fingerprint
• x5t#256 (optional) - boolean flag to include the sha256 fingerprint
• typ (optional) - included from the role definition, possibly support setting custom types in the future?
• cty (optional) - allow setting of a content-type
• crit (optional)
  

  GET /jose/roles/:name

Returns the details about the role

GET /jose/roles/:name/public

Same as the /jose/keys/:name/public just a shortcut if you only know the role

GET /jose/roles/:name/public/raw

Same as the /jose/keys/:name/public/raw just a shortcut if you only know the role

GET /jose/roles/:name/jrl (Jose/JWT Revocation List)

Note: Not sure if this will get implemented, added it to the SPEC as a request, I think it would be lower on the priority for implementation, all feedback welcome though.

Thought: Valid for JWT only?

Returns a JWT token signed by the key that is used for the role, in the payload there will be an array of revoked JTI values.

GET /jose/roles/:name/tidy

Removes all known revoked JTI and expired JTI values from the JRL (Jose/JWT Revocation List)

GET /jose/issue/:name

Gets a JWT, JWS, JWE token from a defined role.

Note: JWT tokens should allow for renewal if exp date is included, if included the secret becomes renewable (perhaps with a grace period like PKI?).

POST /jose/issue/:name

For types that allow for setting values at issue time like JWT, using POST along with parameters

Note: JWT tokens should allow for renewal if exp date is included, if included the secret becomes renewable (perhaps with a grace period like PKI?). With POST since you can set unique data in the JWT token, vault should store this data securely so that if a renewal is called the same data is used to create an updated JWT token with a new expiration date.

References

• https://tools.ietf.org/html/draft-ietf-oauth-json-web-token-32
• https://tools.ietf.org/html/draft-ietf-jose-json-web-encryption-40
• https://tools.ietf.org/html/draft-ietf-jose-json-web-signature-41
• https://tools.ietf.org/html/draft-ietf-jose-json-web-key-41

[adamlc]

This looks good! With JWT, vault is automatically generating a JTI (JWT ID) as a UUID, I guess it would be down to the developer to create a system to deal with revocations? Is that something we want to handle in vault?

[ekristen]

We could do a CRL like the PKI backend does but unlike PKI there is no defined CRL like structure, it could just be a list of JTIs that have been revoked.

[adamlc]

That could work! Devs could then just keep a local cache of the revoked JTIs 👍

[gilgoodridge]

JWT Claims are all optional according to the spec. Is there a way when setting up the role (POST /jose/roles/:name) to disable any of the claims listed under "type: jwt - claims"? For example, if I didn't want the JWT to contain the iat, could I do that by including something in the body of the POST to /jose/roles/:name?

[ekristen]

@gilgoodridge I added some clarification. All JWT claims are optional except JTI, I believe JTI should always be set to a UUID. For claims like nbf, iat, exp whether they are included or not should be a boolean value (defaults to true), they are set based on current date and time.

[adamlc]

@ekristen just a quick thought, I assume the allow_custom_payload_at_issue setting is to allow custom claims to be passed for JWTs? Would any custom claims need to be restricted (in respect to roles, for example if we have an admin claim, like admin: true, these may need to be restricted in roles)?

[ekristen]

Admittedly the naming needs some work but one the heartburn items for @jefferai with my original implementation was that vault would just blinding sign anything that was passed in. This would allow if set when defining the role for a custom arbitrary json payload to be passed in that gets added to the JWT token, it wouldn't be able to "override" any of the defined claims unless allow_override_at_issue was set.

[gilgoodridge]

@ekristen I'm wondering why jti shouldn't also be optional since it's optional in the JWT spec? I suspect that you want to make it mandatory to support the JRL feature. But, you could just say that the JRL feature is not supported for tokens that don't contain a JTI.

I'm not entirely sure I understand the JRL feature. It's not clear to me where the revocation list comes from. I thought that there would be an API to provide revoked tokens (JTIs) and those JTIs would be added to the JRL. However, I don't see that in the API... or maybe I'm overlooking it.

[ekristen]

Hey @gilgoodridge great question, here is my reasoning, for background, here is the "mission statement" for vault.

Vault secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. Vault handles leasing, key revocation, key rolling, and auditing.

With that said now, my reasoning is that beyond storing the secrets needed to generate the JOSE related messages, adding particular information like the JTI to ensure audit capabilities, also one thing I forgot to mention in the SPEC which I will add is that I am suggesting that on issue, Vault records the payload so you can "renew" the token to get an updated expiration date (if set), and the data stays the same.

The concept about the JRL (which isn't something I originally planned but added in for comment, I'll make a note in the SPEC to this) is that when you call vault revoke like you would with the PKI backend or any other backend, in this case we can add the JTI to a list like the PKI so that you could check if the JWT has been revoked or not even if the expiration hasn't yet expired.

@gilgoodridge is there a use-case where you would not want the JTI to be present?

[skippy]

hey! A few questions:

• is there a benefit for splitting out keys and roles, vs. just having roles? the PKI splits out certs from roles so you can define a chain of custody; but that isn't needed here is it?
• wouldn't you want to always define sub/aud at POST /jose/issue/:role_name and not at role creation? Or is the idea you can update/modify any claim type at either role creation and/or issue?
• I would choose one way to issue a new credential, and probably just drop GET /jose/issue/:role_name and just use POST; BUT what is interesting is that some of the secrets, like PKI, use POST [secret_type]/issue/... while most seem to use GET [secret_type]/creds. Do you know which direction the vault team wants to go in, from an api consistency standpoint? @jefferai ?
• what about shifting GET /jose/roles/:name/tidy to POST /jose/tidy? this is to bring it inline with other vault secret endpoints like PKI... that way as an operator I can just have a cron job that calls this once a week (or day) rather than having to first retrieve all the roles, iterate through them, and then call this once per role

thanks for leading the charge on this, btw!

[ekristen]

is there a benefit for splitting out keys and roles, vs. just having roles? the PKI splits out certs from roles so you can define a chain of custody; but that isn't needed here is it?

The idea behind this was to follow suite with how transit and pki backends work since we are dealing with a similar structure, this also allows key reuse more easily.

Use-Case: you want one private key to be responsible for signing all JOSE requests as it pertains to your production environment, but you have two different applications with different roles that need to use two different JWT structures (for example), so by creating the key one then creating multiple roles you can more easily break down permissions and usage.

wouldn't you want to always define sub/aud at POST /jose/issue/:role_name and not at role creation? Or is the idea you can update/modify any claim type at either role creation and/or issue?

If you take the concept of authenticating users from a SSO like application, the subject is going to change per user that logs in, so if you had to only define them at role creation time, you'd have to have a role per user, which isn't very user friendly IMO. With this method you get both.

I would choose one way to issue a new credential, and probably just drop GET /jose/issue/:role_name and just use POST; BUT what is interesting is that some of the secrets, like PKI, use POST [secret_type]/issue/... while most seem to use GET [secret_type]/creds. Do you know which direction the vault team wants to go in, from an api consistency standpoint? @jefferai ?

GET is for "read" operations, where as POST is for write operations. Some secret issuance requires you to write (aka POST) data to get the credentials (ie certificate issuance) whereas AWS access keys do not, the just require a read to get them.

what about shifting GET /jose/roles/:name/tidy to POST /jose/tidy? this is to bring it inline with other vault secret endpoints like PKI... that way as an operator I can just have a cron job that calls this once a week (or day) rather than having to first retrieve all the roles, iterate through them, and then call this once per role

Great though, the reason why I initial put this on the role is in thinking that the JRL would be per role vs the entire backend. However there could be room to also add /jose/tidy which would clean up all of them. Definitely something to think about.

thanks for leading the charge on this, btw!

No problem. Glad to do it. Thanks for the feedback @skippy

[gilgoodridge]

Hey @ekristen what you said all makes sense. For the one use-case that I'm currently working on I am passing a JTI in the JWT but it's not used by the receiving/authenticating side (just being validated that's there). So, it turns out that forcing a JTI doesn't hurt my particular use-case. It was more of a curiosity question. And I understand revocation better now. Thanks!

[ekristen]

@gilgoodridge awesome, sounds good, if you have any more comments let me know.

[skippy]

hey @ekristen

The idea behind this was to follow suite with how transit and pki backends work since we are dealing with a similar structure, this also allows key reuse more easily.

Use-Case: you want one private key to be responsible for signing all JOSE requests as it pertains to your production environment, but you have two different applications with different roles that need to use two different JWT structures (for example), so by creating the key one then creating multiple roles you can more easily break down permissions and usage.

but the reason why this is useful for PKI is so there is a centralized chain of trust; ie. I can put one root cert on my servers and the various roles can all be verified against that one CA.

I see the use-case for certs and a central certificate authority, but not for JOSE, unless of course I'm missing something? There is nothing wrong in having this abstraction per se; my point is more around the question of if the added complexity of all the /jose/key endpoints is worth it.

If you take the concept of authenticating users from a SSO like application, the subject is going to change per user that logs in, so if you had to only define them at role creation time, you'd have to have a role per user, which isn't very user friendly IMO.

Exactly; that was my point :) I was wondering if there is a reason to have sub/aud defined at role creation time (esp. sub)? Though having the _consistency_ between role creation and role issue in terms of what options are available is a nice touch.

[ekristen]

@skippy - sorry this ended up being a little long winded even after I rewrote the response 3 times :/

Thanks for the feedback, here's some of my thoughts on it.

I see the use-case for certs and a central certificate authority, but not for JOSE, unless of course I'm missing something? There is nothing wrong in having this abstraction per se; my point is more around the question of if the added complexity of all the /jose/key endpoints is worth it.

It's a tough call, I think it really depends on what algorithms you are using. If you are using symmetric algorithms then abstracting the keys might seem like added complexity, but if you are using asymmetric keys (like I am) then I think the abstraction makes more sense.

This goes back to the previous use case where I have a primary private key (aka CA) responsible for signing JWT tokens for various things, like user login, api access etc. Defining the key once inside vault, then being able to generate different roles that allow for different claims to be set or maybe to be hardcoded, then setting policies to allow support personnel to generate tokens under one set of rules, and microservices or users under another set of rules while just referring back to the single key seems more simple?

I think the key abstraction makes even more sense when and if you are start to enable the jku, kid, x5t and related fields, as all these fields can be populated automatically.

However to your point about complexity -- if a vault admin is only going to generate a single key using a symmetric algorithm (like HS256) and thus a shared secret and a single role then it could definitely look to complex.

Exactly; that was my point :) I was wondering if there is a reason to have sub/aud defined at role creation time (esp. sub)? Though having the consistency between role creation and role issue in terms of what options are available is a nice touch.

Ultimately I think provides a little more flexibility for various use cases I can't think of.

Use Case: Using JWT to authenticate microservices to each other. If you setup roles in vault with defined subjects and audiences that can't be overwritten and then give each microservice policies to generate tokens, then each microservice would be able to generate a specific token to access a specific service's API.

[Ginja]

@ekristen This looks great, and exactly what we're looking to do with Vault!

[skippy]

@ekristen thanks for the detailed writeup.

[ekristen]

@skippy np, I'm working on a PR that'll implement this, hopefully the work won't be rejected.

[jefferai]

@ekristen Looking at go-jose, it seems extremely high level. I've worked with other JWT libs in Go that allow you to define claims and then serialize the remaining object; here it seems like the only actual thing that it does is sign an arbitrary payload. e.g. Signer is an interface that just has a Sign method that works on existing []byte. Am I missing something?

[ekristen]

If there is a better library out there I'm all for it, can you share with me what libraries you've used?

I started a proof of concept branch locally that implements some of the basic primitives like key creation, role creation, and I have a basic issue endpoint working, but nothing that can't be updated.

Thanks @jefferai

[jefferai]

I've been using https://github.com/SermoDigital/jose with success.

Question for you -- I and/or @vishalnayak want to look at this soon -- is the proposal in the top comment updated to reflect the outcome of the conversations that were had in this issue? Would be nice to start with a current state of the world so we're all on the same page.

[ekristen]

Awesome news on you two potentially having time soon!

Yes it should be up to date to reflect what was discussed, I'll review it again in the next day or so and make sure there isn't anything that needs to be updated.

Erik

Sent from my iPhone

On Jan 11, 2017, at 13:34, Jeff Mitchell [e-mail] wrote:

I've been using https://github.com/SermoDigital/jose with success.

Question for you -- I and/or @vishalnayak want to look at this soon -- is the proposal in the top comment updated to reflect the outcome of the conversations that were had in this issue? Would be nice to start with a current state of the world so we're all on the same page.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[adamlc]

Any updates on this? 😃

[fproulx-pbox]

👍 👍 this would be amazing to support JWT / IdToken / OpenID etc.

[adamlc]

This might be a good candidate for the new plugin system 👍

[wenisman]

i wrote a basic JWT backend which started off following this proposal however I changed it to suit my purposes. Happy to change to suit the proposal, PM me if this anyone is still interested in following up on this

https://github.com/wenisman/vault_jwt_plugin

[robwilkes]

Had this exact thought today, wouldn't it be great if the key to generate my JOSE (specifically JWE) was stored in a vault, never left the vault, however that would require the vault to understand JOSE formatted messages. Alas someone has already proposed it!

I would love to see this!

[jefferai]

Note that @wenisman's plugin is for an _auth_ backend, while this issue is about a _secrets_ backend.

[wenisman]

Yep I had to change the proposal to suit my needs, (aka making a login endpoint) however it's not too say it can't be changed to meet the proposal.

Again happy to go ahead and make it meet the proposal if there's a need for it

[techninja1008]

Any progress?

[wenisman]

Is this for me or just the plug-in in general?

[centromere]

Is there someone actively working on this? If so, is there a way to track progress?

[ekristen]

TBQH, I’ve lost motivation.

I will not be helping with this.

Erik

On Jun 22, 2018, at 14:09, John Galt [e-mail] wrote:

Is there someone actively working on this? If so, is there a way to track progress?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[jefferai]

Yes, someone implemented this: https://github.com/immutability-io/jwt-auth

We've reached out to the author to discuss upstreaming, but haven't heard back yet.