Microsoft-authentication-library-for-dotnet: [Feature Request] Device code flow ADFS 2019 Support

[keystroke]

Is your feature request related to a problem? Please describe.

Using MSAL with ADFS 2019 is missing some features and needs design clarifications:

• [Topic 1] Device login flow is disabled in the MSAL code, but ADFS supports this.

• [Topic 2] Clarification on behavior for scopes to use when acquiring app tokens.

• [Topic 3] Usage patterns for code that must interact with AAD or ADFS (cloud applications that can be deployed in Azure Stack need to support either AAD or ADFS); current approach requires the caller to determine if target environment is AAD or ADFS, and then call either .WithAuthority or .WithAdfsAuthority which kind of ruins the whole fluent-style of coding. Is it possible that MSAL could just detect ADFS authorities in .WithAuthority option directly, including knowing to ignore .WithTenant for ADFS login API, as well as ignoring authority validation? More details on this behaver in ADAL repo which has similar problem for consuming code: ADAL #1653

Describe the solution you'd like
Discussion on topics raised for ADFS integration and how client code should handle using MSAL for both AAD and ADFS dynamically.

Describe alternatives you've considered
~

Additional context

Regarding scopes, This documentation link shows and discusses using "/.default" for the target STS to "understand" to which scopes this should map to for the calling application. In AAD, this has semantic meaning for AAD to translate this literal scope quest into the "real" set of scopes which are statically configured for the application via it's manifest. For ADFS, no such "./default" scope mapping feature exists, though some work could be done by each individual ADFS customer to customize and add this scope and add additional claims transference rules to each application to translate the ".default" literal scope value into an "actual" set of scope values.

Above is explicitly a problem for Azure PowerShell folks, who are using MSAL with ADFS-backed Azure Stack instances, and other variations of private cloud devices. They have added this same code and value for ADFS code path, but ADFS does not understand what "./default" means, nor does it know how to map it dynamically for each application (without significant configuration overhead, especially since it has no real permission model).

We are wondering specifically what value should Azure PowerShell use here when using MSAL client for ADFS, or in general what you'd advise for these ADFS code paths. It seems 'openid' scope is always supported OOTB for all ADFS environments, which might make for a better "fallback / default" value to use for the scope when requesting tokens. Specifically for Azure Stack, the current services are ignoring scope values in tokens, and only looking at audience field for validation, so technically it would seem that "any" valid scope understood by ADFS could be used here. But for Azure PowerShell, there are various cmdlets that will call various services, each of which requires a token for potentially different resource / scope value (e.g. if any Azure services start using scope-based auth, this approach in Azure PS of a static claim value wouldn't work very well).

How should PowerShell team setup this code here for determining scope values for calling applications? Any advice? My thought would be for AzurePS to have an internal mapping of "resource to scope value" (e.g. the KeyVault cmdlets in Azure PowerShell would be responsible to "determine" the scope value to be used when acquiring tokens against the KeyVault service, and so rather than Azure PowerShell designing a single static value to always be used, they would need a way for the cmdlet (which has context on what token needs to be acquired) to provide the target scope value (at least for ADFS case)). The current documentation and behavior for AAD has led them to design their use of MSAL to always use "./default" for all service principal authentication, but it seems like this might need to be more dynamic for ADFS scenario? Or perhaps they can just use 'openid' for now? What is your take on this?

[keystroke]

@bganapa, @markcowl

[bganapa]

Thanks @keystroke for describing the issue in details, @jmprieur please advise

[jmprieur]

@keystroke @bganapa @markcowl
[Topic 1] - @trwalke do you remember why device code flow is currently disabled for ADFS? maybe it has not been released yet when we started supporting ADFS 2019?
[[Topic 2] - I don't think that you should ever use .default, except for client credentials (where it's compulsory). My understanding is that AzurePS is acquiring tokens on behalf of the user? and therefore, if the resources are v1.0 resources, I would assume that the corresponding scope would be "$[resource/user_impersonation}". That might change in the future, though, when Web API start adding more scopes, so adding a mapping might be a good idea.
Topic 3] - We'll come back to you for .WithAuthority(). cc: @henrik-me @jennyf19

[keystroke]

@jmprieur Regarding [Topic 2], no we are talking about the client credentials flow (Azure PS lets user authenticate with a service principal for writing scripts, this uses the client credentials flow) and using a scope value of "default" it is not supported in ADFS (you would have to implement your own thing in ADFS to achieve the same effect as what is done in AAD, as ADFS does not have the same permission model, and has no understanding of ".default"). I also don't think it's "compulsory" in AAD scenario either (AFAIK is an "optional shortcut" to keep the specifics out of your code, but I could be wrong about that).

[trwalke]

@jmprieur yes, the ADFS device code flow support was not ready at the time of our release. However, I spoke with the ADFS team and the said that it should have been released to the public around the third week of august. Ill run tests to see if it is working with the latest version of ADFS 2019. if so then we can enable it for the next release of MSAL

[jmprieur]

@keystroke : for AAD, .default is compulsory when one uses client credentials. All other scopes are rejected by EVO: See https://docs.microsoft.com/en-us/azure/active-directory/develop/scenario-daemon-acquire-token?tabs=dotnet#did-you-use-the-resourcedefault-scope and https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow#first-case-access-token-request-with-a-shared-secret

then for ADFS, you'd have to pass-in all the application scopes?
@yordan-msft : how would you suggest we do?

[keystroke]

@jmprieur Thanks for the link! It looks like this "/.default" thing is specifically for client credentials flow when requesting v1 resource-based tokens from AAD IDP; if they want v2 scope-based tokens then they just use the scopes they want / need for that request.

[Unrelated banter] I assume the resulting access token returned from AAD would have all the correct scopes populated though? How does that affect your caching of the token? Seems MSAL couldn't know whether scopes have been updated in AAD permission model and thus wouldn't know whether they can use the cached token or not for client credentials flow, since the resulting scopes change independent of what the client is asking for. I guess client just lives with ~hour delay until change takes effect, or manually clears cache? Probably not a real problem in most cases except for initial cloud application deployments where services will initially get tokens before any permissions are granted, those won't work and would be cached for a while?

[markcowl]

@jmprieur @keystroke @isra-fel It seems that .default scope is required for client credentials (returning a token effectively scoped with all granted permissions) but that this is not supported in ADFS? What is the ADFS alternative? Note that it is not possible when authenticating for a script to know in advance which permissions are needed, so we need some solution that will simply authenticate any requests based on the permissions of the SPN to that resource.

Also, just as a performance matter, adding specific scopes to tokens based on the action of a single cmdlet seems unworkable - this would involve many more token requests than currently, as the scopes change from cmdlet to cmdlet. It also seems redundant and unnecessary to provide specific scopes when making token requests for a client, as that is what the user already did when they granted [permissions to the client in the first place.

For user credentials, in ADFS, is user_impersonation an available scope for every resource? Note that scripts will need access to not only ARM, but also resources like Graph, KeyVault, Storage, etc.

[keystroke]

@markcowl I do see user_impersonation is understood by ADFS in Azure Stack; not sure if that's by default or if it was added as custom configuration in AS:

Name Description ---- ----------- aza Scope allows broker client to request primary refresh token. user_impersonation Request permission for the application to access the resource as the signed in user. winhello_cert The winhello_cert scope allows an application to request Windows Hello credentials encoded as c... email Request the email claim for the signed in user. profile Request profile related claims for the signed in user. logon_cert The logon_cert scope allows an application to request logon certificates, which can be used to ... openid Request use of the OpenID Connect authorization protocol. allatclaims Requests the access token claims in the identity token. vpn_cert The vpn_cert scope allows an application to request VPN certificates, which can be used to esta...

[bganapa]

@trwalke Can you please let us know when the fix for enabling device code auth be in MSAL? When is the next release?

[jennyf19]

@jmprieur @henrik-me - is this something to include in next release? Please update the milestone. thx.

[trwalke]

@jennyf19 @henrik-me @jmprieur I spoke with @bganapa offline and this is quite urgent. We should consider a smaller release sooner. Ill follow up offline

[bganapa]

@markcowl Could you please confirm whether the Az autogenerated Preview modules would take a dependency on MSAL? i.e is it going to take a dependency on az.accounts 1.6.2 which is built on ADAL or 2.0.0-preview version of az.accounts which built on MSAL. Could you please confirm? @trwalke depending on the answer to the above Q, we need to decide the urgency. Thanks Travis

[henrik-me]

@bganapa @markcowl : Our scheduled release is end of next week. How does that timeline work?

[bganapa]

Thanks @henrik-me, @trwalke End of next week is perfect. I just confirmed with @markcowl that end of next week is a good time frame. Right now azure powershell consumes 2.7.0 version of MSAL. Can we get the fix in that major version so that there are no breaking changes for azure powershell to consume?

[jmprieur]

@bganapa @markcowl
The support for ADFS 2019 was added starting in MSAL 4.0.0. See MSAL.NET 4.0 released / ADFS 2019.
Due to the amount of changes internally between 2.7 and now, I don't think that this would be reasonable to add ADFS to 2.7.

In practice, the migration to 4.x is faciliated by obsolete attributes with actionable messages and aka.ms links (best viewed in VS Output window) which will tell you exactly what to do. If needed you can get a feel with page: Migrating from MSAL 2.x to MSAL 3.x and above

[bganapa]

Thanks @jmprieur for the quick response. @markcowl Can we move on to 4.0* in azure powershell?

[keystroke]

@jmprieur @trwalke @henrik-me

We did some additional testing today.

For user interactive flows, MSAL seems to include some additional scopes into the request by default:
````
$c.AcquireTokenInteractive(string[]).ExecuteAsync().GetAwaiter().GetResult()

(True) MSAL 4.0.0.0 MSAL.Desktop Microsoft Windows NT 10.0.17763.0 [10/02/2019 21:12:42 - c7bc8009-828e-455a-94a4-af45b2b9edf8] (UnknownClient: 0.0.0.0) Navigating to 'https://adfs.redmond.ext-n42r1904.masd.stbtest.microsoft.com/adfs/oauth2/authorize/?
scope=https[:]//adminmanagement.adfs.n42r1904.masd.stbtest.microsoft.com/aad1d07a-c603-4e6e-9fb2-8bbedb09410b/anythingiwant offline_access openid profile
&response_type=code&client_id=1950a258-227b-4e31-a9cf-717495945fc2&redirect_uri=urn:ietf:wg:oauth:2.0:oob&client-request-id=c7bc8009-828e-455a-94a4-af45b2b9edf8&x-client-SKU=MSAL.Desktop&x-client-Ver=4.0.0.0&x-client-CPU=x64&x-client-OS=Microsoft Windows NT 10.0.17763.0&prompt=select_account&code_challenge=ZPQ2kr-R6nEcU99mrgARmBIyTYFLLcqkReepQO47J4k&code_challenge_method=S256&state=[REDACTED]'.
````

Above, you see that we have one scope provided, but MSAL adds a few other scope values into the request. If we try to set those ourselves, MSAL throws an exception and says that it will always add these scopes and they are necessary for the tool to function:

Exception calling "GetResult" with "0" argument(s): "MSAL always sends the scopes 'openid profile offline_access'. They cannot be suppressed as they are required for the library to function. Do not include any of these scopes in the scope parameter."

ADFS has the behavior where at least one request scope needs to be valid, and ADFS will issue a token with that valid scope. So for interactive flows, we are able to use the resource/scope flow to get a token that contains the other scopes (profile and openid); ADFS will see the /.default scope which it doesn't recognize, but it does recognize the other scopes, and so the resulting access token will contain only the recognized scopes, and ADFS just needs there to be at least one recognized scope (and for a target resource to be resolved). So by including the scope of "resource/scope" it informs ADFS of the target audience value for the token, and for the scope values, we get whatever scopes it recognizes (as long as it recognizes at least one). And of course, ADFS must recognize the resource from the resource/scope value, or you'll get the error indicated by the final two examples below:

Scope given to MSAL | Scope in resulting ADFS access token
-- | --
'https://adminmanagement.adfs.n42r1904.masd.stbtest.microsoft.com/aad1d07a-c603-4e6e-9fb2-8bbedb09410b/.default' | profile openid
'https://adminmanagement.adfs.n42r1904.masd.stbtest.microsoft.com/aad1d07a-c603-4e6e-9fb2-8bbedb09410b/user_impersonation' | profile openid user_impersonation
'https://adminmanagement.adfs.n42r1904.masd.stbtest.microsoft.com/aad1d07a-c603-4e6e-9fb2-8bbedb09410b/ ' (empty space at the end) | profile openid
'https://adminmanagement.adfs.n42r1904.masd.stbtest.microsoft.com/aad1d07a-c603-4e6e-9fb2-8bbedb09410b/' | profile openid
'https://adminmanagement.adfs.n42r1904.masd.stbtest.microsoft.com/aad1d07a-c603-4e6e-9fb2-8bbedb09410b' or 'https://adminmanagement.adfs.n42r1904.masd.stbtest.microsoft.com/aad1d07a-c603-4e6e-9fb2-8bbedb09410b//' | MSIS9605: The client is not allowed to access the requested resource.

When we try the same thing for the service principal flow, MSAL does not include the additional claims into the request:
````powershell
$c.AcquireTokenForClient(string[]).ExecuteAsync().GetAwaiter().GetResult()

(True) MSAL 4.0.0.0 MSAL.Desktop Microsoft Windows NT 10.0.17763.0 [10/04/2019 23:14:47 - da3ed8ae-5ab5-4740-80d7-4d8f2a296f9b] (UnknownClient: 0.0.0.0) === Token Acquisition (Client CredentialRequest) started:
Authority: https://adfs.local.azurestack.external/adfs/
Scope: https://adminmanagement.adfs.azurestack.local/2dbfaa0c-db32-4c56-be20-ec8245939950/.default
ClientId: b6459e7b-9e46-48b8-ba97-3d7d6c4b620e
Cache Provided: True

Exception calling "GetResult" with "0" argument(s): "MSIS9605: The client is not allowed to access the requested resource."
````

Above error is an error from ADFS not understanding the ".default" scope. If we try to do the same workaround that is done in the user interactive flow, MSAL will actually prevent us from doing so:

````powershell
$c.AcquireTokenForClient(string[]).ExecuteAsync().GetAwaiter().GetResult()

Exception calling "GetResult" with "0" argument(s): "MSAL always sends the scopes 'openid profile offline_access'. They cannot be suppressed as they are required for the library to
function. Do not include any of these scopes in the scope parameter."
````

Above exception is thrown locally; it seems MSAL is saying "hey I am going to include openid scope for you already, so don't include it yourself" but in the actual request to ADFS for service principal flow, no additional scopes are included.

What is hopeful about this, is that if we could make MSAL include these scopes (which it seems to indicate that it will) then the flow will actually work for ADFS (we tested in fiddler):

````
POST https://adfs.local.azurestack.external/adfs/oauth2/token/ HTTP/1.1
x-client-SKU: MSAL.Desktop
x-client-Ver: 4.0.0.0
x-client-CPU: x64
x-client-OS: Microsoft Windows NT 10.0.17763.0
client-request-id: 24351091-9240-4e07-b871-325023fae7f0
return-client-request-id: true
x-app-name: UnknownClient
x-app-ver: 0.0.0.0
Accept: application/json
Content-Type: application/x-www-form-urlencoded
Host: adfs.local.azurestack.external
Content-Length: 260
Expect: 100-continue
Connection: Keep-Alive

client_id=b6459e7b-9e46-48b8-ba97-3d7d6c4b620e&client_info=1&client_secret=[REDACTED]&scope=https%3A%2F%2Fadminmanagement.adfs.azurestack.local%2F2dbfaa0c-db32-4c56-be20-ec8245939950%2F.default+openid&grant_type=client_credentials

---

HTTP/1.1 200 OK
Cache-Control: no-store
Pragma: no-cache
Content-Length: 1387
Content-Type: application/json;charset=UTF-8
Server: Microsoft-HTTPAPI/2.0
client-request-id: 24351091-9240-4e07-b871-325023fae7f0
Strict-Transport-Security: max-age = 31536000
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:;
Date: Fri, 04 Oct 2019 23:21:57 GMT

{"access_token":"[REDACTED]","token_type":"bearer","expires_in":3600,"scope":"openid"}
````

So our first question is: is this a bug in MSAL, where you meant to include the additional scopes for the service principal client credentials flow? Or is it a bug with an incorrect error message?

And one final question, separate but related: we noticed that AAD seems to accept a redirect_uri parameter for any identity application of http://localhost - we tried with many identity applications we created, including ones with no reply addresses configured, and in all cases, AAD was fine to accept redirect URIs for any port on http://localhost.

And it seems that MSAL will use this fact to use http://localhost to handle user interactive flows by launching the users browser, and attempt to listen on port 8400 by default, though it seems to use the first port it can find for this purpose.

Is that understanding correct? The issue here is that ADFS does not have a global rule to allow these reply addresses, nor can it allow a redirect URI on any arbitrary port. They have to configure each application redirect URI, and they don't support wildcards for the chosen port, meaning we have to specific the exact port(s) that will be used by the client.

Does MSAL decide on what port to use in this kind of flow? Or does the consuming code have to pick a port and ensure it is available for use? For the current tools supported by ADFS on Azure Stack (Azure CLI, AzurePS, etc.) we have http://localhost:8400 added as a valid redirect URI. Should we be adding a few dozen redirect URIs here for various ports that may be used? What is the design and flow of this mechanism, and how should it be setup for ADFS?

For purposes of reply:

• [Issue 1] MSAL scope parameters for various flows
• [Issue 2] MSAL redirect_uri on http://localhost:8400 and port selection / app registration

[bgavrilMS]

@keystroke - I will answer on issue 2 first. I tried to document this here: https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki/System-Browser-on-.Net-Core

But essentially you are correct - ADFS does not seem to support "localhost any port" like AAD does.
So your application will need to do port management on its own - I suggest you "reserve" a few ports, register redirect URIs, and then cycle through them until you find a free one. Then configure MSAL to use it. @reed1995 - does ADFS plan to support this feature to be at parity with AAD ?

[bgavrilMS]

@keystroke - for issue 1, I think your investigation into scopes added by MSAL is not the root cause. MSAL will add some scopes to ensure it can cache tokens. For client credentials, there is no "openid" scope, because there is no ID token involved. There is also no "emai", because the service principal does not have email ...

I believe the issue is that you need to pre-authorize the application. During app registration, you need to add the scope you want and then also "Grant Authorization".

[keystroke]

@bgavrilMS For issue 1, I was asking about the error message. MSAL client is throwing an exception if I provide 'openid' scope, and saying that it will supply it instead. For delegated flows, it actually does this, including the extra scope in the request. For client credential flows, it does not. So either the error message should say "MSAL is not going to let you do that for client credential flows", or it should be updated to not throw and let you do it (perhaps just for ADFS case). Regarding the "conceptual invalidity" of asking for scope 'openid' using client credentials flow in the context of ADFS, I'm not sure whether this is "legitimate" or not and what MSAL should do in this case. ADFS configuration around this is a bit more "open" AFAIK and so having the MSAL client refuse to honor the requested scope entirely might be a bit overkill.

It sounds like the simplest solution (regarding scopes) would be for Azure PS to use "/.default" for all scenarios EXCEPT client credential flows for ADFS environments, and in those cases, use "/openid" which does work as I just tested (MSAL prevents use of 'openid' on it's own but will allow "/openid" and ADFS understands openid scope so resulting access token has correct audience claim, and scp claim of 'openid'). In the context of Azure Stack services, scope fields are "irrelevant" as services only validate on "aud" claim, but in general ADFS doesn't know how to "re-map" a claim of "/.default" into an array of claims, so for ADFS all the claims would need to be requested manually, or advanced ADFS configuration would be required to learn how to re-map '/.default' into a custom list of scopes, but AFAIK that could only be done per resource, not per-client like in AAD, so this flow is fundamentally different anyway.

[Issue 2] Regarding redirect_uri, AFAIK no new features are planned for ADFS at all, so unlikely a change in behavior would be introduced at this point. I believe Azure PS and AzureCLI are using the default port (by omitting it), which I guess MSAL interprets port 80 as an indication to start using random ports on the machine? Is there a flow to which ports it tries? Requiring all the Azure Tooling to implement port management in a cross-platform way might be a bit much, and for the short term they are likely going to be using the default behavior. Can you clarify the port selection algorithm that MSAL uses? Seems like it starts on port 8400 if its available? We can register a few ports that are used by default for each of these azure tool applications as you recommend.

[bgavrilMS]

[Issue2] - If redirect uri is on port 80, MSAL chooses random ports until one is available. Otherwise, MSAL will use the port that you give it. I've tested this logic on all .net core on all 3 operating systems and it works.

            TcpListener listner = new TcpListener(IPAddress.Loopback, 0);

Code is here

By the way, you do not need a redirect URI for client credential flows.

[bgavrilMS]

For issue #1 we will take a bug:

• On client credential grant, we will ensure we do not add any scopes
• On all flows, we will stop throwing if user passes reserved scopes. Instead, we will merge the scopes with our own.

I plan to look at this now so that we can get it in our next release (scheduled for end of week). As an ugly workaround, you can pass a string like "resource/email openid" in the scope array.

@jmprieur @henrik-me @reed1995 - do we want to go the extra step and translate "/.default" to "openid" ? Would this work ? Or at least throw an exception that ".default" does not work?

[jmprieur]

@keystroke : would it be the right thing to do, in the case of ADFS to translate resource/.default into resource/openid ?

[keystroke]

@jmprieur I don't think MSAL should modify the claim values for ADFS scenario, I was just pointing-out that the flow is different for ADFS around the semantics of "{resource}/.default" and MSAL probably shouldn't interfere (I was originally curious if MSAL actually required the extra scopes as the error message indicated, as if that was the case, it would've worked E2E for us due to the way ADFS is processing scopes, but since MSAL doesn't need the extra scopes in the client credential flow, it's probably 'cleaner' not to include them).

For Azure Stack, none of our services are using scope values, as everything is resource-based (so we don't need a specific scope result, just something that gets the right resource-based token). But for other "arbitrary" ADFS setups, customers can configure applications and scopes in essentially any way they like, so having MSAL modifying scope values (or refusing them) might not be the safest thing, as a customer setup may require some specific, literal scope value that is 'unforseen' by MSAL (I like the idea to merge scope values instead of throwing an exception if one of the 'reserved' ones is supplied).

@bgavrilMS Thanks for the code link! The redirect URI thing is for the delegated flows, not client credential flows as you point out. From code link you gave, I see that the port selection on my machine is not entirely predictable; it seems that Azure PowerShell and AzureCLI and any other tooling hoping to work with ADFS will need to 'decide' on a port or a few ports ahead of time, so they can be registered as explicit redirect URIs within ADFS, and provide those ports to MSAL.

@bganapa please coordinate with @markcowl to ensure Azure PS is attempting to use an explicit port that we can register with ADFS.

````powershell
1..5|%{try{($l=[System.Net.Sockets.TcpListener]::new([ipaddress]::Loopback, 0)).Start();$l.LocalEndpoint.Port}finally{$l.Stop()}}

2439
2440
2441
2442
2443

2584
2585
2586
2587
2588

2603
2604
2605
2606
2607
````

[bgavrilMS]

@keystroke - I've eliminated that verification around scopes, MSAL will now just merge scopes. Will go out with our 4.5 release (in a few days)

I believe we can close this thread?

[jennyf19]

@keystroke This is included in the 4.5.0 release.