Vault: OIDC auth method fails to add users to external groups

[janbrunrasmussen]

Describe the bug
Under the JWT auth method I can create external groups and they are populated with users as they login.
Switching to oidc auth method, with same IDP, and as similar a setup as possible, groups are not populated when user login

To Reproduce
Steps to reproduce the behavior:
I have only tested with the IDP I have at hand (Azure AD).

_What works_:

1. vault auth enable jwt
2. vault write auth/jwt/config oidc_discovery_url='https://sts.windows.net/{tenant}/'
3. vault write auth/jwt/role/test-role policies='dev' bound_audiences='{application registration id}' user_claim='oid' groups_claim='groups' role_type='jwt'
4.

vault write identity/group name='test-group' \
        policies='test-policy' \
        type='external'

5.

vault write identity/group-alias name='{a value from groups_claim}' \
        mount_accessor={jwt accessor} \
        canonical_id='{id from group creation in step 4}'

1. vault write auth/jwt/login role=test-role jwt={jwt which has the group alias name in the groups_claim}
   
   
   
   1. Validate that user is in member of the external group - which it is!
   
   

_What does not work_:

1. vault auth enable oidc
2.

vault write auth/oidc/config oidc_discovery_url='https://sts.windows.net/{tenant}/' \
oidc_client_id='{application registration id}' \
oidc_client_secret='{application secret}' \
default_role=test-role

1. vault write auth/oidc/role/test-role policies='dev' bound_audiences='{application registration id}' user_claim='oid' groups_claim='groups' role_type='oidc' oidc_scopes='Groups.Read.All' allowed_redirect_uris='http://localhost:8200/ui/vault/auth/oidc/oidc/callback' allowed_redirect_uris='http://localhost:8250/oidc/callback'
2. Same as above
3. Same as above
4. vault login -method=oidc role=test-role
5. Validate that user is in member of the external group - which it is not!

Expected behavior
It is expected that the external group is populated with the user.

Environment:

• Vault Server Version (retrieve with vault status): 1.1.0
• Vault CLI Version (retrieve with vault version): Vault v1.1.0 ('36aa8c8dd1936e10ebd7a4c1d412ae0e6f7900bd')
• Server Operating System/Architecture: OSX

Vault server configuration file(s):
Test environment:
docker run --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' -e VAULT_LOG_LEVEL=TRACE -p 8200:8200 vault

Additional context
I was doubting if the group claim was not present, but I ran through the authorize request urls for my own jwt generation and the one from vault, and they are the same, except nonce, state and redirict_uri (obviously), and in the Vault logs I see the groups when I log in:

2019-03-28T09:57:05.705Z [DEBUG] identity: creating a new entity: alias="id:"d8132b79-ce46-99de-dae2-b7b6353796cc" canonical_id:"0aeed425-ad4c-f2fd-69da-fa5d76dbd77b" mount_type:"oidc" mount_accessor:"auth_oidc_a0c1a53b" mount_path:"auth/oidc/" name:" {user_claim}" creation_time:<seconds:1553767025 nanos:705565400 > last_update_time:<seconds:1553767025 nanos:705565400 > namespace_id:"root" "
2019-03-28T09:57:05.706Z [DEBUG] identity: refreshing external group memberships: entity_id=0aeed425-ad4c-f2fd-69da-fa5d76dbd77b group_aliases=["mount_accessor:"auth_oidc_a0c1a53b" name:"[" comma separated list of groups in the claim "]

The only thing I find weird, is if I add groups in claim_mappings, I get this error when logging in:

Error: Ember Data Request GET /v1/auth/oidc/oidc/callback returned a 400 Payload (Empty Content-Type) [object Object]

[ncabatoff]

Hi @janbrunrasmussen,

It sounds like you're running into https://github.com/hashicorp/vault-plugin-auth-jwt/issues/17. I reproduced what you experienced using the auth0 provider. I saw the same creating/refreshing log messages as well. Once I created the external group and alias and linked them using a groups_claim, I would see those log messages followed by

[DEBUG] identity: adding member entity ID to external group: member_entity_id=e4e5b6a0-81b3-08c0-9f66-213c95d45ccd group_id=33a23388-7cf5-4273-7be2-93341f73c15e

See this comment which is how I made sense of the problem and got it working.

[ncabatoff]

Apologies, I didn't read your bug report carefully enough. You already understand the issues I brought up in my previous comment. I'm not sure why it's not working for you. I'll try to reproduce it with Azure.

[janbrunrasmussen]

Thanks for looking at it @ncabatoff! Let me know if I left out relevant details.

[ncabatoff]

Here's a possible hint.

Error: Ember Data Request GET /v1/auth/oidc/oidc/callback returned a 400 Payload (Empty Content-Type) [object Object]

I got this error as well when the groups_claim mapped to a null value in the id token. When I fixed it so that there was an empty list instead, the error disappeared and I was able to login, though without being a member of the group I wanted. When I had it return a list with the single element matching the name of my group alias, I was able to login and was added to the group.

[bartse]

@ncabatoff I'm not sure if I'm running into the same issue or not but I configured the provider in the same way and getting the same error code.

I'm wondering how the group claim and claim mappings would look like. Would it be possible to detail the parameters?

[janbrunrasmussen]

I am getting this error when I set the groups_claim to a value that does not exist as a claim (e.g. thisisnotintheclaim, but when I set groups_claim='groups' the error disappears and I can log in:

Error: Ember Data Request GET /v1/auth/oidc/oidc/callback returned a 400 Payload (Empty Content-Type) [object Object]

The error reappears if I set groups in claim_mappings.
Also, I can see the groups in the logs, so they are extracted from the claim, but again, not matched to the entity.

[ncabatoff]

I have an explanation for why putting groups in claim_mappings results in an error: we require the groups_claim return a list of strings, but the extractMetadata() call that copies claim_mappings into metadata requires that the value of each referenced claim must be a string, and returns an error otherwise, failing the authentication process at the stage where it's trying to create the identity.

Now as to your primary problem, I think I may also have an explanation. You cited your logs:

2019-03-28T09:57:05.706Z [DEBUG] identity: refreshing external group memberships: entity_id=0aeed425-ad4c-f2fd-69da-fa5d76dbd77b group_aliases=["mount_accessor:"auth_oidc_a0c1a53b" name:"[" comma separated list of groups in the claim "]

name should be a single value, not a comma separated list. So maybe the problems is that Azure's Groups.Read.All is returning ["group1,group2"] instead of ["group1", "group2"]? This is what the log line looks for me in the latter case:

2019-03-29T08:43:10.867-0400 [DEBUG] identity: refreshing external group memberships: entity_id=5444ea5b-5ec6-64f0-11b3-50acc68a5b46 group_aliases=["mount_accessor:"auth_oidc_3f1e86f9" name:"admin" ", "mount_accessor:"auth_oidc_3f1e86f9" name:"test" "]

@bartse We're working on improving the documentation. Claims are very provider-specific. I have something working with Auth0 here.

[janbrunrasmussen]

Well, I am unsure how Vault parses the groups, but here is the actual output:

2019-03-28T10:45:57.559Z [DEBUG] identity: refreshing external group memberships: entity_id=0aeed425-ad4c-f2fd-69da-fa5d76dbd77b group_aliases=["mount_accessor:"auth_oidc_a0c1a53b" name:"[\"5a79ca51-24ee-4412-9e11-02720819c6fb\",\"....\",\"540ff2c7-e8da-4228-9a56-ede79a6b58cf\"]" "]

[ncabatoff]

Right, so the problem is a mismatch between the Vault OIDC auth method groups_claim requirements and what Azure is giving us as groups with the Groups.Read.All scope:

groups_claim (string: <optional>) - The claim to use to uniquely identify the set of groups to which the user belongs; this will be used as the names for the Identity group aliases created due to a successful login. The claim value must be a list of strings.

I'm not sure whether we should be asking Azure to provide an equivalent mechanism that returns a list of strings, or trying to accommodate this use case. I don't believe there's any kind of standard to follow here. I'll discuss with the rest of the team.

In the meantime, you might consider reaching out to Azure support and asking if they have any suggestions.

[janbrunrasmussen]

But what I don't understand is why the group mapping works under jwt but not oidc?

[ncabatoff]

What does the jwt token look like?

[kalafut]

I was testing this and saw no groups claims at all in my token when using OIDC. Setting "groupMembershipClaims" to "All" in the application manifest at least resulted in a list of groups in the token. I'm learning this too; this may not be the best/correct way.

ref: https://docs.microsoft.com/en-us/azure/active-directory/develop/reference-app-manifest#manifest-reference

[janbrunrasmussen]

I am not sure how to share to jwt itself without giving away sensitive information, but just to reiterate: Group mapping works with the jwt auth method, but when using the oidc auth method it does not. I am using exactly the same application registration in Azure and I can see the group claim values in the Vault logs. So Vault is getting the groups from the token, but users are not mapped to the groups.

[ncabatoff]

I don't know how you're creating the JWT and I know very little about Azure so it's hard for me to conclude anything from the above. Rather than share the actual JWT token, you could decode it using something like https://jwt.io/ then post the payload here, eliding any sensitive information. I just want to see the structure.

[kalafut]

@janbrunrasmussen One thing I noticed is your oidc_discovery_url being https://sts.windows.net. I think(?) that's the V1 endpoint. My endpoints (i.e. in the portal) are all of the form: https://login.microsoftonline.com/{tenant}/v2.0. Further, when I curl the .well-known endpoint for these two, the STS one has fewer supported scopes.

The issue of V1 vs. V2 tokens was mentioned recently on the mailing list:
https://groups.google.com/d/msg/vault-tool/ZcnDIP71rhk/W6Dq4rtSCQAJ

[janbrunrasmussen]

@ncabatoff here is an example of the jwt:

{
  "aud": "{application registration id}",
  "iss": "https://sts.windows.net/{tenantid}/",
  "iat": 1554118551,
  "nbf": 1554118551,
  "exp": 1554122451,
  "amr": [
    "pwd",
    "mfa"
  ],
  "family_name": "{REDACTED}",
  "given_name": "{REDACTED}",
  "groups": [
    "5a79ca51-24ff-4412-9e11-02720819b6fc",
    "540ff2c7-e4da-4248-9a56-ede39a6b55kf",
    "...",
    "e82cd389-3370-4c2d-ae2f-ddbc218932as",
    "3ba07ae3-cd87-4c7b-bf66-e62224860a24"
  ],
  "ipaddr": "{REDACTED}",
  "name": "{REDACTED}",
  "nonce": "{REDACTED}",
  "oid": "{unique user identifier}",
  "onprem_sid": "{REDACTED}",
  "sub": "{REDACTED}",
  "tid": "{REDACTED}",
  "unique_name": "{REDACTED}",
  "upn": "{REDACTED}",
  "uti": "{REDACTED}",
  "ver": "1.0"
}

Some details: Before oidc auth method was available, we made a small service (based on go-oidc, same as Vault oidc auth method) that would produce a jwt token, so we could use the jwt auth method in Vault. As mentioned multiple times, this works with user mapping to groups :) To find out why the mapping was not working with oidc auth method, I dug into the way Vault gets the jwt, and I can confirm that the authorize call is made to the same endpoint, and with the same parameters: https://login.windows.net/{tenanid}/oauth2/authorize?client_id={application registration id}&nonce={nonce}&redirect_uri={redirect}&response_type=code&scope=openid+Groups.Read.All&state={state}
Therefore I am (maybe not correctly) assuming that the token Vault gets is the same as I am producing with our own service.
This is a V1 token - I will try with a V2 token based on the issue you mention in the mailing list, but it comes with it's own issues (don't get me started..), so I would prefer if it worked with V1.

[kalafut]

That JWT looks pretty reasonable... One point I'd like to confirm: when you're referring to the working JWT approach, is that with the current version of Vault, or something earlier? The group alias extraction code is common to both jwt/oidc (https://github.com/hashicorp/vault-plugin-auth-jwt/blob/9474f90fb1df957c622f30d26eaa2f5f6a1b89b5/path_login.go#L262), which makes this all the more odd. Just wondering if we missed something when refactoring code into this method.

[janbrunrasmussen]

Yes! The repro steps were been run on the same Vault version (https://github.com/hashicorp/docker-vault/blob/e6353cca743f951019a46a1cc597c34d9ff24f38/0.X/Dockerfile):

docker run --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' -e VAULT_LOG_LEVEL=TRACE -p 8200:8200 vault

Given the fact I can see the groups in the Vault logs, I assume this is were we end up: https://github.com/hashicorp/vault-plugin-auth-jwt/blob/master/path_login.go#L307
The input claim could differ from jwt and oidc:
https://github.com/hashicorp/vault-plugin-auth-jwt/blob/9474f90fb1df957c622f30d26eaa2f5f6a1b89b5/path_login.go#L84
https://github.com/hashicorp/vault-plugin-auth-jwt/blob/9474f90fb1df957c622f30d26eaa2f5f6a1b89b5/path_login.go#L245

[kalafut]

@janbrunrasmussen I think we've finally gotten to the bottom of what's causing this. When logging in via OIDC, the ID token is parsed, and then any claims found at the /userinfo endpoint are merged in. When using the sts endpoint the, groups list in the ID token is fine, but the same claim also exists in the /userinfo response as the stringified list: "groups":["[\"9e58c006-42ba-4406-9b13-f6132b0b29ab\",\"352b6bbe-9f59-4a65-9a7e-2f6cd2f26494\",\"45dd570d-2706-4d0a-bea7-b09fb4ad5691\"]"]. That value is replacing the good groups claim, which is why we're see this issue only in the OIDC+//sts... case.

So that's what's going on. We're still reviewing the best approach here. Thanks for your help and patience!

[janbrunrasmussen]

I looked at that code block for a bit yesterday, but skipped passed it as I figured that would only have the profile, but of cause it has all the data. Why the data would differ from the token is a different issue - you gotta love Azure AD for that..
We are still waiting for a V2 application registration, so I cannot validate whether that will work for us or not, but I am assuming so based on the mailing list issue above. Otherwise we know which piece of code we need to comment out :)

In any case, thanks you so much @kalafut for being super helpful and responsive! Please keep us updated with your decisions on this.

[janbrunrasmussen]

Finally got to test with a V2 application registration and it works! It did require a bit of meddling with the scope though.
If it has interest, I can document further details here (or elsewhere)? https://www.vaultproject.io/docs/auth/jwt_oidc_providers.html#azure-active-directory-aad-

[kalafut]

Updates to that document would be much appreciated!

I’d also like to know more generally whether your new v2 registration still allows you to do everything you need with its v1 endpoint in other contexts. From the docs it seems like that’s the case, but I’m not familiar enough with it to know what the limitations are, if any.

[kalafut]

@janbrunrasmussen I'm going to close _this_ issue now since we've isolated the cause to some surprising AD behavior (i.e. not a more general external groups bug). You're more than welcome to open a new issue(s) regarding AD handling if necessary. The topic of how to handle provider quirks is a larger issue for its own thread.

Thanks for your help working through this and for the docs updates.