Azure-functions-host: System.MissingMethodException: Method not found: Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfiguration.get_SigningKeys()

[prabh-62]

Repro steps

Provide the steps required to reproduce the problem:

• Ensure you have Azure Functions V3 func CLI from Github releases and .NET Core 3.1 SDK
• > dotnet --version
3.1.401
• > func --version
3.0.2798
• > mkdir Assembler and > cd Assembler
• > dotnet new classlib -n Assembler.Lib55 (Create new class library)
  
  
  
  • cd Assembler.Lib55
  
  
  • dotnet add package Microsoft.IdentityModel.Protocols.OpenIdConnect --version 5.5.0
  
  
  • Create a class named Lib55Class with following code snippet
    
    csharp namespace Assembler.Lib55 { public class Lib55Class { public async Task ReturnNothing() { var metadataAddress ="https://justformajidstests.b2clogin.com/justformajidstests.onmicrosoft.com/B2C_1_DUMMY/v2.0/.well-known/openid-configuration"; var configurationManager = new ConfigurationManager<OpenIdConnectConfiguration>( metadataAddress, new OpenIdConnectConfigurationRetriever(), new HttpDocumentRetriever() ); var config = await configurationManager.GetConfigurationAsync(CancellationToken.None).ConfigureAwait(false); var signingKeys = config.SigningKeys; } } }
  
  

• > dotnet new classlib -n Assembler.Lib56 (Create new class library)

  • cd Assembler.Lib56
  • dotnet add package Microsoft.IdentityModel.Protocols.OpenIdConnect --version 5.6.0
  • Create a class named Lib56Class with similar code snippet
    csharp namespace Assembler.Lib56 { public class Lib56Class { public async Task ReturnNothing() { var metadataAddress ="https://justformajidstests.b2clogin.com/justformajidstests.onmicrosoft.com/B2C_1_DUMMY/v2.0/.well-known/openid-configuration"; var configurationManager = new ConfigurationManager<OpenIdConnectConfiguration>( metadataAddress, new OpenIdConnectConfigurationRetriever(), new HttpDocumentRetriever() ); var config = await configurationManager.GetConfigurationAsync(CancellationToken.None).ConfigureAwait(false); var signingKeys = config.SigningKeys; } } }

• > func init Assembler.Functions --dotnet

  • Update nuget Microsoft.NET.Sdk.Functions to 3.0.9 and add MSBuild property <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput> to Assembler.Functions.csproj
  • Add project reference to both class libraries. > dotnet add reference ../Assembler.Lib55 and dotnet add reference ../Assembler.Lib56

• Create a new class Debug.cs with this code snippet

namespace Assembler.Functions
{
    public static class Debug
    {
        [FunctionName(nameof(Debug))]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "debug")] HttpRequest req,
            ILogger log)
        {
            var c55 = new Lib55Class();
            await c55.ReturnNothing().ConfigureAwait(false);

            //NOTE: If the following line is removed/commented out, the exception may not always happen!
            var assembly = Assembly.GetAssembly(typeof(OpenIdConnectConfiguration));

            var c56 = new Lib56Class();
            await c56.ReturnNothing().ConfigureAwait(false);
            return new OkResult();
        }
    }
}

• > func start and invoke API http://localhost:7071/api/debug
  

Expected behavior

HTTP Status code should be 200

Actual behavior

HTTP Status is 500 with an exception

System.Private.CoreLib: Exception while executing function: Debug. Assembler.Lib55: Method not found: 'System.Collections.Generic.ICollection`1<Microsoft.IdentityModel.Tokens.SecurityKey> Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfiguration.get_SigningKeys()'.

Known workarounds

None

Related information

Provide any related information

• Programming language used: C#
• Bindings used: HTTPTrigger

[prabh-62]

By the way, I also created ASP.NET Core API to verify if the underlying issue is with ASP.NET Core as well or just Azure Functions.
> dotnet new webapi -n Assembler.AspnetCore

And I added the same code in WeatherForecastController that I added to Azure Functions earlier

namespace Assembler.AspnetCore.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class WeatherForecastController : ControllerBase
    {
        [HttpGet]
        public async Task<IActionResult> Get()
        {
            var c55 = new Lib55Class();
            await c55.ReturnNothing().ConfigureAwait(false);

            //NOTE: If the following line is removed/commented out, the exception may not always happen!
            var assembly = Assembly.GetAssembly(typeof(OpenIdConnectConfiguration));

            var c56 = new Lib56Class();
            await c56.ReturnNothing().ConfigureAwait(false);
            return new OkResult();
        }
    }
}

I couldn't reproduce the same issue in ASP.NET Core 3.1 project

[prabh-62]

My colleague, Majid (@themajix) created a github repo with more details.

Repo: https://github.com/themajix/mutiversion_assemblies_in_functions

[prabh-62]

• We have a fairly big Azure Functions project (140 functions)
• In around ~130 azure functions, we use nuget Microsoft.AspNetCore.Authentication.JwtBearer 3.1.7 to authorize the HTTP calls with JWT token and this nuget requires Microsoft.IdentityModel.Protocols.OpenIdConnect 5.5.0
• In 1 azure function, we use nuget EFCore.BulkExtensions 3.1.5 to bulk insert into Azure SQL and this nuget requires Microsoft.IdentityModel.Protocols.OpenIdConnect 5.6.0

[prabh-62]

What we have observed so far is

• In the case of success, we see OpenIdConnect and its dependant assemblies loaded in FunctionAssemblyLoadContext
  

• However, in case of exception, we only see OpenIdConnect assembly in FunctionAssemblyLoadContext
  

Not sure if FunctionAssemblyLoadContext loads assemblies with its dependencies or just loads the requested assembly?

cc @fabiocav and @brettsam as they are the code owners for FunctionAssemblyLoadContext.

[brettsam]

This is an awesome repro, thank you! I'm looking at it now.

[brettsam]

Let me post my notes here as I spent quite a while investigating it:

This can happen under the following circumstances:

1. The function assembly is referencing other assemblies, which are referencing other assemblies that are also used by the host
   
   
   
   • One assembly references the assembly with the same version (or early) version as the host
   
   
   • One assembly references the same assembly, but with a higher version than the host
   
   
2. When someone requests a host assembly that is equal or lower, we always return the Host's version -- we call this "assembly unification" but it's similar to Binding redirects. However, higher-versioned assemblies would fall outside of this and be loaded into the Function's LoadContext.
3. When the assembly referencing the same version makes a Load request, we see that this is one of our host assemblies and happily return the host's version from it's LoadContext (in this case, 5.5.0)
4. However, we don't realize yet that the Function's LoadContext is about to request a newer version of that assembly and we should be returning it instead. But by then, the damage is done.

If 3 and 4 happen in the opposite order, we'll correctly return 5.6.0 first and everything works. So we need a way to make sure we always return that assembly.

I have a prototype fix ready but it'll need some review but it will need some more testing before merging -- I've scheduled this for our next Sprint.

FYI -- In this specific case, you end up with a type mis-match b/c the SigningKeys property returns ICollection<SigningKey> -- and we've already returned Microsoft.IdentityModel.Tokens.SigningKey 5.5.0, yet the Function LoadContext now expects 5.6.0 -- and since the type versions don't match, the methods don't match.

[brettsam]

Oh I should note -- tomorrow morning I'm going to investigate a workaround for you. It may involve you force-loading types from these "higher-versioned" assemblies early in your process.

And thank you IMMENSELY for this repro -- this is a complicated scenario and you made it a lot simpler to work with. I really appreciate that.

[themajix]

Thank you so much Brett! We really appreciate how fast you responded to our request. I'm sure that this will make the whole team here really happy, as we have been struggling with this issue for a while now.

[brettsam]

To set expectations, this likely won't be released for roughly a month (with the release that begins at the end of our next Sprint). I can, however, provide you with a private site extension that you can test on your own before that time and let me know if it works.

There's also a workaround in the meantime. You need to explicitly load the troublesome assemblies from your Function early, to ensure that the correct version is loaded. You can do this with a FunctionsStartup class like this:

using FunctionTests;
using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.IdentityModel.Logging;
using Microsoft.IdentityModel.Protocols;
using Microsoft.IdentityModel.Protocols.OpenIdConnect;
using Microsoft.IdentityModel.Tokens;

[assembly: FunctionsStartup(typeof(Startup))]

namespace FunctionTests
{
    public class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            var t = typeof(SecurityKey); // Microsoft.IdentityModel.Tokens
            t = typeof(OpenIdConnectConfiguration); // Microsoft.IdentityModel.Protocols.OpenIdConnect
            t = typeof(LogHelper); // Microsoft.IdentityModel.Logging
            t = typeof(AuthenticationProtocolMessage); // Microsoft.IdentityModel.Protocols
        }
    }
}

This forced version 5.6.0 of these assemblies to be loaded and everything seemed to work. If you want to take this approach-- let me know and I can help you identify whether there are other assemblies that could lead to this.

[themajix]

Yes, I think we are going to take this approach to minimize the risk of having runtime errors in our production environment. Do you already have a utility to identify potentially problematic assemblies in a project, or will the identification task be a manual one?

[brettsam]

We're going to move this to Sprint 85 as there are already 2 other AssemblyLoadContext issues going in this sprint and we'd rather spread them out to minimize any negative impact. The workaround will continue to work in the meantime.

[brettsam]

Because we're bundling Sprint 84 and 85 together, we unfortunately have to move this to Sprint 86 now.

[pragnagopa]

PR is in progress. Leaving this assigned to Sprint 86.

[helgeu]

Just hit this problem. Downgraded from 6.8.0 (latest of the openid packages) to 5.5.0. That alone did NOT fix the problem. Added the above typeof-code and then it did work.

[helgeu]

I was a bit quick in the previous comment. It did in fact always work locally. And it did work in ONE of the test environments we have. Not the most important test environment though, so its a bit more random than expected even.

[helgeu]

Not entirely confirmed yet, but I had to make sure all our startup classes had the above "fix" in them. If not then it was a bit random who "won" and what was in effect... At least when I did put the "fix" into both our startup-classes it worked as expected again.