Aspnetcore: Version mismatches in 2.1 and 2.2 patch updates (often causes FileLoadException)

[natemcmaster]

Some ASP.NET Core 2.1 users have been affected by a bug in the .NET Core host (see https://github.com/dotnet/core-setup/issues/4512) which can cause apps to fail with System.IO.FileLoadException. The issue may also be present in your app, even if the app does not fail with System.IO.FileLoadException.

Affected versions

.NET Core, 2.1.1 through 2.1.4 (all OSes), 2.1.7 (Windows only), 2.2.1 (Windows only)

Symptom 1

ASP.NET Core apps running on the Microsoft.AspNetCore.App shared framework may fail to run after the machine-wide version of .NET Core is updated. Apps fail with an error such as:

Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.Identity.Core, Version=2.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
at Diary.Api.Startup.ConfigureServices(IServiceCollection services)

Symptom 2

When publishing an ASP.NET Core application which references Microsoft.AspNetCore.App, additional Microsoft assemblies are in the publish output.

For example, if you run dotnet publish --output PublishDir/ and you see Microsoft.Extensions.*.dll or Microsoft.AspNetCore.*.dll files in PublishDir/, you may be running into this issue.

Cause

This issue is typically caused by .csproj files which have a PackageReference to a package which is already part of Microsoft.AspNetCore.App.

For example,

<ItemGroup>
  <PackageReference Include="Microsoft.AspNetCore.App" />
  <PackageReference Include="Microsoft.AspNetCore.SignalR" Version="1.0.3" />
</ItemGroup>

The Microsoft.AspNetCore.SignalR package is already pulled in as a part of Microsoft.AspNetCore.App, so does not normally need to be referenced. The presence of this reference causes a mismatch of versions, which in turn causes a problem with the .NET Core host, as described in https://github.com/dotnet/core-setup/issues/4376.

More technical details here: https://github.com/Azure/app-service-announcements-discussions/issues/65#issuecomment-417379227

Workaround 1

Remove PackageReference's which are already part of Microsoft.AspNetCore.App. You can find a list of what this package pulls in by looking at https://www.nuget.org/packages/microsoft.aspnetcore.app#show-dependency-groups. This will avoid the version mismatch.

Workaround 2

Upgrade the Microsoft.AspNetCore.App PackageReference to the latest 2.1.x version by adding the Version attribute.

  <PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.3" />

:warning: This may have unintended consequences on your deployment environment. If the deployment environment does not have the the 2.1.3 runtime, apps will fail to start with an error. In Azure or apps hosted behind IIS, this may appear as HTTP 502.5 error.

[mholec]

I found issue similar to symptom 2, but with some differences:

I have ASP.NET Core application (Microsoft.AspNetCore.App 2.1.2) FDD deployed to Azure App Service (West Europe). Everything worked fine for three weeks but 19/09/2018 all of the sudden it starts to fail. I tryied dotnet tool on App Service (Kudu) and there is only App/All runtime version 2.1.1 and 2.1.4 on the machine.

My csproj file contains:

    <PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.2" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="2.1.2" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.4.1" /> 

and it is failing on this error:

Application: dotnet.exe CoreCLR Version: 4.6.26814.3 Description: The process was terminated due to an unhandled exception. Exception Info: System.IO.FileLoadException: Could not load file or assembly 'Microsoft.EntityFrameworkCore, Version=2.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040) at DotnetTalks.Startup.<>c__DisplayClass7_0.<ConfigureServices>b__3(DbContextOptionsBuilder opt) at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.<>c__DisplayClass1_0`2.<AddDbContext>b__0(IServiceProvider p, DbContextOptionsBuilder b) at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.DbContextOptionsFactory[TContext](IServiceProvider applicationServiceProvider, Action`2 optionsAction) at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.<>c__DisplayClass10_0`1.<AddCoreServices>b__0(IServiceProvider p) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider) at DotnetTalks.Program.Main(String[] args) in D:\a\1\s\DotnetTalks\Program.cs:line 15
</Data>

I tryied to change the file this way:

    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="2.1.2" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.4.1" /> 

but the error remains. When I explicitly change Sqlite version this way:

<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="2.1.2" />

it works. But Microsoft.EntityFrameworkCore.Sqlite is not a part of ASP.NET Core runtime (App).

It is very unpleasant to know that the application can whenever fall down. I found that only safe solution is to deploy application as self-contained. Any ideas how to figure out this issue?

[natemcmaster]

@mholec did you try Workaround 2?

[mholec]

I can try it. However the problem is that this kind of error means that my web application my be broken at any time after successful deploy and test.

[natemcmaster]

We have a fix for the issue in 2.1.5 which should be out soon, so only 2.1.1 through 2.1.4 should be affected by this issue. When 2.1.5 is released, I recommend you update your servers.

[natemcmaster]

I have verified the fix is in 2.1.5 which will be released soon.

Fixed with https://github.com/dotnet/core-setup/pull/4521

[hobbify]

I recently experienced this problem, only for adding more info my problems was that the runtime was higher than my references so like the solution from giorgio I only updated the EF.Tools vía nuget

[natemcmaster]

This issue has resurfaced in 2.1.7. We are investigating the cause. I believe the workarounds mentioned above should continue to work to resolve this.

@hobbify in your case, try Workaround 2.

The cause of the problem now is an incomplete .deps.json file in the Microsoft.AspNetCore.App shared framework. It does not contain all the information it needs to about assembly versions. For example, look at C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.1.7\Microsoft.AspNetCore.App.deps.json

By contrast, 2.1.6 has the right metadata on "runtime" entries.

        "runtime": {
          "lib/netstandard2.0/Microsoft.AspNetCore.Mvc.Core.dll": {
            "assemblyVersion": "2.1.3.0",
            "fileVersion": "2.1.3.18263"
          }
        }

In 2.1.7, this metadata is missing.

        "runtime": {
          "lib/netstandard2.0/Microsoft.AspNetCore.Mvc.Core.dll": {}
        }

[frankkilcommins]

@natemcmaster For workaround 2 what version do you recommend? From comparing different app services created during our deployments different run-time versions are available. I can try with 2.1.6

[natemcmaster]

Wherever possible, I recommend the latest, 2.1.7, but it depends on your hosting server. The version you specify will be your application's minimum. If your hosting server doesn't have ASP.NET Core >= version, your app fail with "It was not possible to find any compatible framework version".

[natemcmaster]

Proposed fix => https://github.com/aspnet/AspNetCore/pull/6619. It needs to go through approval channels first, but assuming that happens, this will be fixed in 2.1.8 and 2.2.2.

[axelheer]

After installing Version 2.1.8 I'm running into issues with apps compiled using:

<ItemGroup>
  <PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.7" AllowExplicitVersion="true" />
</ItemGroup>

Installing the update on our servers would make them all crash with:

Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'Microsoft.AspNetCore, Version=2.1.7.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
   at Program.CreateWebHostBuilder(String[] args)

We stopped the rollout of the newest update for now, but

• How can we run applications compiled using 2.1.7 when there's 2.1.8 additionally (!) installed?
• How can we handle this in the future? Not specifying an explicit version leads to many Microsoft.Extensions.*.dll or Microsoft.AspNetCore.*.dll in the publish directory, which leads to other version hell issues and bloats our build artifacts...

[jeremyVignelles]

Same issue here, using

<PackageReference Include="Microsoft.AspNetCore.All" Version="2.1.7" />

When removing the explicit version, the app works, but I'm afraid of the impact of unwanted updates. If we remove the version, will it always stay in the 2.1.x branch?

I'm fine with having an explicit version here since I'm redistributing the runtime with my app (dotnet publish -c Release -r win-x64)

[axelheer]

Okay, using Version 2.1.6 instead of 2.1.7 seems to work with 2.1.7 and 2.1.8. Welcome to .NET Core Version Hell.

Thus, we need to downgrade our apps to 2.1.6 and wait with the rollout of 2.1.8 until every rollout of our apps is done. Does that make sense?

Is that really necessary or is there a better way?

[axelheer]

Ad Workaround 1) That does not help, if an app references some non Microsoft package, which references some Microsoft.AspNetCore.* package.

[natemcmaster]

@axelheer Installing the update on our servers would make them all crash

I've opened https://github.com/aspnet/AspNetCore/issues/7546 to resolve the 2.1.7 -> 2.1.8 issue which causes FileLoadException.

You are correct that Workaround 1 does not work if you depend on other things which pull in AspNetCore transitively packages. We're addressing this in 3.0 with a fundamental redesign of the way we version and distribute aspnetcore packages and assemblies. In the meantime, use Workaround 2.

@jeremyVignelles: When removing the explicit version, the app works, but I'm afraid of the impact of unwanted updates. If we remove the version, will it always stay in the 2.1.x branch?

By default, .NET Core will run on the highest patch version of 2.1.* installed on the machine, regardless of the presence/absence of the Version attribute on the PackageReference to AspNetCore.App. Version should really have been named MinimumVersion. This is a common pitfall, so you're not the first to ask.

When you remove the explicit version attribute and build a self-contained application (dotnet publish -r win-x64), the .NET Core SDK will attempt to give you the latest version of AspNetCore.App. To get a newer version of aspnetcore in a self-contained app, you should either (1 - recommended) update the SDK or (2 - as a workaround) set the <RuntimeFrameworkVersion> property to the version you want to use.

If you want to further control the way in which your application rollsforward (or does not rollforward) to new versions, checkout instructions here: https://natemcmaster.com/blog/2019/01/09/netcore-primitives-3/#automatically-run-on-higher-patch-versions

[replaysMike]

It would seem this is still an issue with .Net Core 3.0

[natemcmaster]

@replaysMike please open a new issue with more details about your problem. This one has been closed for a long time.