Azure-sdk-for-js: DefaultAzureCredential fails on Azure App Service when using User-Assigned Identity

[noelbundick]

• Package Name: @azure/identity
• Package Version: 1.1.0 / 1.2.0-beta.1
• Operating system: Linux
• [x] nodejs
  
  
  
  • version: v12.13.0
  
  
• [ ] browser
  
  
  
  • name/version:
  
  
• [ ] typescript
  
  
  
  • version:
  
  
• Is the bug related to documentation in
  
  
  
  • [ ] README.md
  
  
  • [ ] source code documentation
  
  
  • [ ] SDK API docs on https://docs.microsoft.com
  
  

Describe the bug

When using DefaultAzureCredential on an Azure App Service that has only a User-Assigned Managed Identity, the call to getToken() fails with an exception and does not continue to the next entry in the chain, causing an unhandled exception in user code, despite being properly configured

To Reproduce
Steps to reproduce the behavior:

1. Create an Azure Web App with the Linux container image ghcr.io/noelbundick/credential-bug-repro
   
   
   
   • Corresponding code here: https://github.com/noelbundick/credential-bug-repro
   
   

1. Assign a User-Assigned identity only

System identity disabled

One user-assigned identity

1. Set the identity's clientId as the AZURE_CLIENT_ID app setting

User assigned identity clientId

AZURE_CLIENT_ID in Application Settings

1. Run the following commands from your local machine to witness the failure. You can see the exception details in the App Service Log stream (screenshots below). You may need to enable application file system logging on the site

• curl https://<myapp>.azurewebsites.net/repro1 - fails despite setting the managedIdentityClientId option
• curl https://<myapp>.azurewebsites.net/repro1 -fails despite having AZURE_CLIENT_ID properly configured

Expected behavior

I expect to be able to use DefaultAzureCredential as documented. I expect the initial usage of ManagedIdentityCredential without a clientId set to fail, and for the chain to continue down and use the one that is configured to use my User-Assigned Identity.

Additional context
Stack trace from App Service w/ AZURE_LOG_LEVEL=verbose below. You can see the url being called by ManagedIdentityCredential does not include the clientId.

2020-10-01T22:37:14.635Z INFO  - Container mitestapp_1_8c954fb0 for site mitestapp initialized successfully and is ready to serve requests.
2020-10-01T22:36:59.890091434Z Hosting environment: Production
2020-10-01T22:36:59.890825444Z Content root path: /app
2020-10-01T22:36:59.891290350Z Now listening on: http://[::]:8081
2020-10-01T22:36:59.891301850Z Application started. Press Ctrl+C to shut down.
2020-10-01T22:37:50.740308426Z azure:identity:info EnvironmentCredential => Found the following environment variables: AZURE_CLIENT_ID
2020-10-01T22:37:50.742838359Z azure:core-http:info ServiceClient: using custom request policies
2020-10-01T22:37:50.743701971Z azure:core-http:info ServiceClient: using custom request policies
2020-10-01T22:37:50.744603182Z azure:core-http:info ServiceClient: using custom request policies
2020-10-01T22:37:50.746190703Z azure:identity:info EnvironmentCredential => getToken() => ERROR: EnvironmentCredential is unavailable. Environment variables are not fully configured.
2020-10-01T22:37:50.747910726Z azure:identity:info ManagedIdentityCredential => Using the endpoint and the secret coming form the environment variables: MSI_ENDPOINT=http://172.16.2.6:8081/msi/token and MSI_SECRET=[REDACTED].
2020-10-01T22:37:50.749844552Z azure:identity:info IdentityClient: sending token request to [http://172.16.2.6:8081/msi/token?resource=https%3A%2F%2Fmanagement.azure.com&api-version=2017-09-01]
2020-10-01T22:37:50.753594101Z azure:core-http:info Request: {
2020-10-01T22:37:50.753630001Z   "url": "http://172.16.2.6:8081/msi/token?resource=REDACTED&api-version=2017-09-01",
2020-10-01T22:37:50.753635701Z   "method": "GET",
2020-10-01T22:37:50.753639802Z   "headers": {
2020-10-01T22:37:50.753643502Z     "_headersMap": {
2020-10-01T22:37:50.753647302Z       "accept": "application/json",
2020-10-01T22:37:50.753651202Z       "secret": "REDACTED",
2020-10-01T22:37:50.753655102Z       "accept-language": "REDACTED",
2020-10-01T22:37:50.753658902Z       "x-ms-client-request-id": "8bba1e26-2750-463a-8ec7-3f70f04a987e",
2020-10-01T22:37:50.753662802Z       "content-type": "application/json; charset=utf-8",
2020-10-01T22:37:50.753666502Z       "user-agent": "core-http/1.1.9 Node/v12.13.0 OS/(x64-Linux-4.15.0-112-generic)"
2020-10-01T22:37:50.753670402Z     }
2020-10-01T22:37:50.753674002Z   },
2020-10-01T22:37:50.753677402Z   "query": {
2020-10-01T22:37:50.753681002Z     "resource": "REDACTED",
2020-10-01T22:37:50.753690202Z     "api-version": "2017-09-01"
2020-10-01T22:37:50.753694202Z   },
2020-10-01T22:37:50.753697702Z   "withCredentials": false,
2020-10-01T22:37:50.753701302Z   "timeout": 0,
2020-10-01T22:37:50.753704802Z   "keepAlive": true,
2020-10-01T22:37:50.753708402Z   "requestId": "8bba1e26-2750-463a-8ec7-3f70f04a987e"
2020-10-01T22:37:50.753712202Z }
2020-10-01T22:37:50.978830270Z azure:core-http:info Response status code: 400
2020-10-01T22:37:50.979770082Z azure:core-http:info Headers: {
2020-10-01T22:37:50.979786282Z   "_headersMap": {
2020-10-01T22:37:50.979790782Z     "content-type": "application/json; charset=utf-8",
2020-10-01T22:37:50.979861983Z     "date": "Thu, 01 Oct 2020 22:37:50 GMT",
2020-10-01T22:37:50.979867683Z     "server": "Kestrel",
2020-10-01T22:37:50.979882584Z     "transfer-encoding": "chunked"
2020-10-01T22:37:50.979942984Z   }
2020-10-01T22:37:50.979946784Z }
2020-10-01T22:37:50.983336129Z azure:identity:warning IdentityClient: authentication error. HTTP status: 400, An unknown error occurred and no additional details are available.
2020-10-01T22:37:50.984070639Z azure:identity:info ChainedTokenCredential => getToken() => ERROR: ManagedIdentityCredential authentication failed.(status code 400).
2020-10-01T22:37:50.984085239Z More details:
2020-10-01T22:37:50.984096339Z unknown_error(status code 400).
2020-10-01T22:37:50.984100039Z More details:
2020-10-01T22:37:50.984103439Z An unknown error occurred and no additional details are available.
2020-10-01T22:37:50.987213880Z (node:41) UnhandledPromiseRejectionWarning: AuthenticationError: ManagedIdentityCredential authentication failed.(status code 400).
2020-10-01T22:37:50.987228080Z More details:
2020-10-01T22:37:50.987232680Z unknown_error(status code 400).
2020-10-01T22:37:50.987236381Z More details:
2020-10-01T22:37:50.987239981Z An unknown error occurred and no additional details are available.
2020-10-01T22:37:50.987243381Z     at ManagedIdentityCredential.<anonymous> (/app/node_modules/@azure/identity/dist/index.js:1077:23)
2020-10-01T22:37:50.987247181Z     at Generator.throw (<anonymous>)
2020-10-01T22:37:50.987289681Z     at rejected (/app/node_modules/tslib/tslib.js:112:69)
2020-10-01T22:37:50.987294181Z     at processTicksAndRejections (internal/process/task_queues.js:93:5)
2020-10-01T22:37:50.989992617Z (node:41) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
2020-10-01T22:37:50.990475023Z (node:41) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Inlined examples from the linked repo so readers don't have to browse the code. These usages of DefaultAzureCredential should work, but they don't:

// Fails: explicitly set AZURE_CLIENT_ID
app.get('/repro1', async (req, res) => {

  const cred = new identity.DefaultAzureCredential({
    managedIdentityClientId: process.env.AZURE_CLIENT_ID
  });

  // This call will take ~120s and ultimately throw
  await cred.getToken('https://management.azure.com/.default'); 
  res.send('OK!');
});

// Fails: let DefaultAzureCredential handle it via env var
app.get('/repro2', async (req, res) => {

  const cred = new identity.DefaultAzureCredential();

  // This call will take ~120s and ultimately throw
  await cred.getToken('https://management.azure.com/.default');
  res.send('OK!');
});

cc @jongio - we should probably validate this scenario for other languages as well

[noelbundick]

Note: I believe this will be resolved via https://github.com/Azure/azure-sdk-for-js/pull/11426

[sadasant]

@noelbundick Thank you, Noel! We'll most likely be releasing today. I'll send you instructions a bit later. After that, if you have the time to try again, please let us know. Otherwise, I'll go more in depth later this week.

[sadasant]

@noelbundick Hello Noel, we have released @azure/[email protected]! Please install it and try again. I'm planning to dive deeper into this issue in the following days, but please let us know if that version helps.

[msftbot[bot]]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[jongio]

@noelbundick - Did this resolve this issue for you?

[noelbundick]

@sadasant sorry for the delay. Still broken for me w/ the same issue.

The initial ManagedIdentityCredential call fails as expected, but it doesn't cascade down and try what should be the next credential - the one that uses my User-Assigned Identity

2020-10-20T22:48:54.074709545Z azure:identity:info EnvironmentCredential => Found the following environment variables: AZURE_CLIENT_ID
2020-10-20T22:48:54.078261668Z azure:core-http:info ServiceClient: using custom request policies
2020-10-20T22:48:54.078890272Z azure:core-http:info ServiceClient: using custom request policies
2020-10-20T22:48:54.085245513Z azure:core-http:info ServiceClient: using custom request policies
2020-10-20T22:48:54.086743022Z azure:identity:info EnvironmentCredential => getToken() => ERROR: EnvironmentCredential is unavailable. Environment variables are not fully configured.
2020-10-20T22:48:54.088141731Z azure:identity:info ManagedIdentityCredential => Using the endpoint and the secret coming form the environment variables: IDENTITY_ENDPOINT=http://172.16.0.2:8081/msi/token and IDENTITY_HEADER=[REDACTED].
2020-10-20T22:48:54.096042082Z azure:identity:info IdentityClient: sending token request to [http://172.16.0.2:8081/msi/token?resource=https%3A%2F%2Fmanagement.azure.com&api-version=2017-09-01]
2020-10-20T22:48:54.106157347Z azure:core-http:info Request: {
2020-10-20T22:48:54.106172247Z   "url": "http://172.16.0.2:8081/msi/token?resource=REDACTED&api-version=2017-09-01",
2020-10-20T22:48:54.106178347Z   "method": "GET",
2020-10-20T22:48:54.106182347Z   "headers": {
2020-10-20T22:48:54.106186147Z     "_headersMap": {
2020-10-20T22:48:54.106230248Z       "accept": "application/json",
2020-10-20T22:48:54.106235448Z       "x-identity-header": "REDACTED",
2020-10-20T22:48:54.106239848Z       "accept-language": "REDACTED",
2020-10-20T22:48:54.106243848Z       "x-ms-client-request-id": "ae721c1c-e662-4253-bf80-2f7e62f76380",
2020-10-20T22:48:54.106247648Z       "content-type": "application/json; charset=utf-8",
2020-10-20T22:48:54.106251548Z       "user-agent": "core-http/1.1.9 Node/v12.13.0 OS/(x64-Linux-4.15.0-112-generic)"
2020-10-20T22:48:54.106255548Z     }
2020-10-20T22:48:54.106259148Z   },
2020-10-20T22:48:54.106262648Z   "query": {
2020-10-20T22:48:54.106266348Z     "resource": "REDACTED",
2020-10-20T22:48:54.106270148Z     "api-version": "2017-09-01"
2020-10-20T22:48:54.106273848Z   },
2020-10-20T22:48:54.106277248Z   "withCredentials": false,
2020-10-20T22:48:54.106280948Z   "timeout": 0,
2020-10-20T22:48:54.106284548Z   "keepAlive": true,
2020-10-20T22:48:54.106288248Z   "requestId": "ae721c1c-e662-4253-bf80-2f7e62f76380"
2020-10-20T22:48:54.106292148Z }
2020-10-20T22:48:54.658892703Z azure:core-http:info Response status code: 401
2020-10-20T22:48:54.667002855Z azure:core-http:info Headers: {
2020-10-20T22:48:54.667022156Z   "_headersMap": {
2020-10-20T22:48:54.667077856Z     "content-length": "0",
2020-10-20T22:48:54.667084356Z     "date": "Tue, 20 Oct 2020 22:48:53 GMT",
2020-10-20T22:48:54.667089056Z     "server": "Kestrel"
2020-10-20T22:48:54.667093556Z   }
2020-10-20T22:48:54.667097656Z }
2020-10-20T22:48:54.667101556Z azure:identity:warning IdentityClient: authentication error. HTTP status: 401, An unknown error has occurred. Response body:
2020-10-20T22:48:54.667105856Z 
2020-10-20T22:48:54.667109556Z 
2020-10-20T22:48:54.667747260Z azure:identity:info ChainedTokenCredential => getToken() => ERROR: ManagedIdentityCredential authentication failed.(status code 401).
2020-10-20T22:48:54.667771260Z More details:
2020-10-20T22:48:54.667775560Z unknown_error(status code 401).
2020-10-20T22:48:54.667820461Z More details:
2020-10-20T22:48:54.667825161Z An unknown error has occurred. Response body:
2020-10-20T22:48:54.667828861Z 
2020-10-20T22:48:54.667832361Z 
2020-10-20T22:48:54.668939368Z (node:47) UnhandledPromiseRejectionWarning: AuthenticationError: ManagedIdentityCredential authentication failed.(status code 401).
2020-10-20T22:48:54.668953368Z More details:
2020-10-20T22:48:54.668957568Z unknown_error(status code 401).
2020-10-20T22:48:54.668961168Z More details:
2020-10-20T22:48:54.668964768Z An unknown error has occurred. Response body:
2020-10-20T22:48:54.668968468Z 
2020-10-20T22:48:54.669010168Z 
2020-10-20T22:48:54.669015068Z     at ManagedIdentityCredential.<anonymous> (/app/node_modules/@azure/identity/dist/index.js:1237:23)
2020-10-20T22:48:54.669019268Z     at Generator.throw (<anonymous>)
2020-10-20T22:48:54.669022968Z     at rejected (/app/node_modules/tslib/tslib.js:112:69)
2020-10-20T22:48:54.669026769Z     at processTicksAndRejections (internal/process/task_queues.js:93:5)
2020-10-20T22:48:54.669294370Z (node:47) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
2020-10-20T22:48:54.675198908Z (node:47) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

[noelbundick]

I believe the root cause is in the error handler here: https://github.com/Azure/azure-sdk-for-js/blob/ff0ac514bcadd0791c9ed855312c18e526932838/sdk/identity/identity/src/credentials/managedIdentityCredential.ts#L481

I suspect this is yet another case of IMDS does one thing, and AppService does another (or maybe AppService does different things based on whether it gets passed the secret / x-identity-header headers??). It's clear that the failure case here is throwing up 401 instead of 400 for me, but then it goes unhandled and we throw up an AuthenticationError instead of a CredentialUnavailable. When this happens, it kills the handling in ChainedTokenCredential and blows up my app even after we seemingly fixed it.

I notice that my original stack trace returned a 400 and the secret header. The newer one returns 401 and uses x-identity-header. Seem like a possible culprit.

Not sure what else this may affect, but the fix might just be err.statusCode === 400 || err.statuscode === 401
By throwing directly, it short-circuits the entire CTC chain and kills the app without continuing.

[noelbundick]

I think there's some additional unusual behavior in the handline of api-version going on here. Clearly we're being supplied with IDENTITY_ENDPOINT and IDENTITY_HEADER as we see in the logs. The handling here seems to use that to explicitly translate that we're using api-version 2019-08-01 (which is in this case incorrect): https://github.com/Azure/azure-sdk-for-js/blob/35b68e4e760a8f91b0eaa9e707ddb97216a1a625/sdk/identity/identity/src/credentials/managedIdentityCredential.ts#L271

The actual query params that get generated are just always hardcoded to 2017-09-01 regardless - so that would seem like the SDK is generating improper URLs:

• https://github.com/Azure/azure-sdk-for-js/blob/35b68e4e760a8f91b0eaa9e707ddb97216a1a625/sdk/identity/identity/src/credentials/managedIdentityCredential.ts#L123
• https://github.com/Azure/azure-sdk-for-js/blob/35b68e4e760a8f91b0eaa9e707ddb97216a1a625/sdk/identity/identity/src/credentials/managedIdentityCredential.ts#L25

On top of that, per the docs for Managed Identity on App Service - 2019-08-01 isn't available for Linux Consumption apps (my test case here is one). Version 2017-09-01 is required and that still uses the secret header instead of x-identity-header: https://docs.microsoft.com/en-us/azure/app-service/overview-managed-identity?tabs=dotnet#using-the-rest-protocol

I'm not sure which thing is causing the 400/401/etc. Addressing the root cause may be more involved than just adding 401 to the error list

[JasonBeneteau]

@noelbundick Your on to something here...

There is a bug in these lines:

https://github.com/Azure/azure-sdk-for-js/blob/35b68e4e760a8f91b0eaa9e707ddb97216a1a625/sdk/identity/identity/src/credentials/managedIdentityCredential.ts#L134

https://github.com/Azure/azure-sdk-for-js/blob/35b68e4e760a8f91b0eaa9e707ddb97216a1a625/sdk/identity/identity/src/credentials/managedIdentityCredential.ts#L148

I believe they should be queryParameters: {...queryParameters, "api-version": version},

[sadasant]

@noelbundick & @JasonBeneteau thank you so much for that detailed response! I'll be able to follow up soon.

[sadasant]

@noelbundick @JasonBeneteau Thank you for your feedback! I'm working on this right now.

[sadasant]

@noelbundick @JasonBeneteau I have a solution! I'll try to write you over teams first.

[sadasant]

Alright, so I confirmed with @noelbundick that the latest version of the Identity package, with my refactoring changes, solves this problem. I'm going to spin some time digging up what exactly fixes this from the refactoring, then I'll comment again, and I'll reference the PR that fixes it.

[sadasant]

I'm running out of time today for studying what exactly fixes this, but my guess is that the parameters get polluted on the current code. The PR that fixes this is: https://github.com/Azure/azure-sdk-for-js/pull/11976

[sadasant]

@noelbundick & @JasonBeneteau hi hi! I've merged the fix. I'll reach out to you for a final test this week, then I'll reply back once the package is finally released.

[sadasant]

@noelbundick & @JasonBeneteau We've released Identity 1.2.0 here: https://www.npmjs.com/package/@azure/identity/v/1.2.0

Please install it and try again 🌞