Aspnetcore: Upgrading Visual Studio to 16.4 broke the execution of an In-Process asp.net core 3.0.1 application. Error 500.30 ANCM

[saysmaster]

Describe the bug

Just after upgrading to VS 2019 16.4, i cannot execute (CTRL+F5) one of my asp.net core 3.0 projects (3.0.1 actually) (using IISExpress). I did not upgrade the project to 3.1.

Other solutions on 500.30 error did not work for me

Update 1

Executing with dotnet run, succesfully starts up the application at localhost:5000

Update 2

When changed the settings in the debug section of the project, I chose IISExpress and _out of process_ model, it executes succesfully with CTRL+F5

Update 3

After fiddling around I came up with these observations:

1. I build the application in my local pc. In my pc there is installed .net core SDK 3.1. I publish to a local folder, and then I xcopy the files to the actual server were the site is hosted under IIS. In the server there is installed only the .net core 3.0 run-time. The site fails to run on the server with the same 500.30 ANCM error.
2. I edit the web config on the site on the server and switch to "OutOfProcess". No other change. The binaries are the same that failed before. This time the site starts up successfully.
3. I put a global.json file in my solution folder and specify SDK version 3.0.100. I built again, publish to local folder and xcopy to server. First observation is that this time the total size of files to xcopy is different even though both times I targeted .netcoreapp3.0. I would expect that the size is the same. Changing the SDK to use, changes the actual assemblies to include or reference??
4. On the server, I switch back to InProcess in web.config. Now I have the assemblies published with the 3.0.100 SDK from my pc. The site starts successfully even InProcess.

So when I build with latest SDK (3.1) the site fails to start InProcess even in a machine with only 3.0 run-time installed.
When I build with specified SDK 3.0.100 the site starts successfully on server machine.

In every case, the project is netcoreapp3.0, not 3.1

Initial error output

Output in browser:

HTTP Error 500.30 - ANCM In-Process Start Failure
Common causes of this issue:
The application failed to start
The application started but then stopped
The application started but threw an exception during startup
Troubleshooting steps:
Check the system event log for error messages
Enable logging the application process' stdout messages
Attach a debugger to the application process and inspect
For more information visit: https://go.microsoft.com/fwlink/?LinkID=2028265

When debugging (F5) the application executes the ConfigureServices method, but it doesn't enter the Configure method

I have some swashbuckle nuget packages referenced in rc versions. I thought they were the problem but even when I removed all references the project still did not start.

Executing the exact same codebase from another pc with vs 2019 16.3.10 installed did not have any problems.

The only exception I could see from Debug console is this:
Exception thrown: 'System.NullReferenceException' in System.Private.CoreLib.dll

This exception did not happen in the vs16.3 execution.

Uninstalling the 3.1 hosting bundle did not solve the problem. The application is supposed to use 3.0.1 right?

To Reproduce

I'm not sure what is needed to reproduce. Other 3.0.1 projects run fine.

Further technical details

dotnet --info:

.NET Core SDK (reflecting any global.json):
 Version:   3.1.100
 Commit:    cd82f021f4

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.100\

Host (useful for support):
  Version: 3.1.0
  Commit:  157910edee

.NET Core SDKs installed:
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  3.0.100 [C:\Program Files\dotnet\sdk]
  3.1.100 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Output from debug when I Debug (F5) the project (only the last lines)

'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.1\System.Text.Encodings.Web.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.1\System.Reflection.Emit.Lightweight.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\3.0.1\Microsoft.Extensions.Localization.Abstractions.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.1\System.Data.Common.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.1\System.Console.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.1\System.Threading.Thread.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Microsoft.Extensions.Hosting.Internal.Host: Debug: Hosting starting
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.1\Microsoft.Win32.Registry.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.1\System.Xml.XDocument.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.1\System.Private.Xml.Linq.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.1\System.Private.Xml.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: Information: User profile is available. Using 'C:\Users\thanasis\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.
The thread 0x2fcc has exited with code 0 (0x0).
Exception thrown: 'System.NullReferenceException' in System.Private.CoreLib.dll
The thread 0x95c has exited with code 0 (0x0).
The program '[14572] iisexpress.exe' has exited with code -1 (0xffffffff).

Visual Studio 2019 16.4

Project .csproj (almost)

<Project Sdk="Microsoft.NET.Sdk.Web">
...
    <PropertyGroup>
        <TargetFramework>netcoreapp3.0</TargetFramework>
        <AspNetCoreHostingModel>InProcess</AspNetCoreHostingModel>
        <AspNetCoreModuleName>AspNetCoreModuleV2</AspNetCoreModuleName>
        <PublishTrimmed>false</PublishTrimmed>
        <PublishSingleFile>false</PublishSingleFile>
        <Nullable>enable</Nullable>
    </PropertyGroup>

    <PropertyGroup>
        <GenerateSerializationAssemblies>Off</GenerateSerializationAssemblies>
        <GenerateDocumentationFile>true</GenerateDocumentationFile>
        <NoWarn>$(NoWarn);1591</NoWarn>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.0.0" />
        <PackageReference Include="Microsoft.AspNetCore.Authorization" Version="3.0.0" />
        <PackageReference Include="Microsoft.AspNetCore.Hosting" Version="2.2.7" />
        <PackageReference Include="Microsoft.AspNetCore.Server.IIS" Version="2.2.6" />
        <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="3.0.1" />
        <PackageReference Include="Microsoft.Extensions.Hosting" Version="3.0.1" />
        <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="3.0.0" />
        <PackageReference Include="Swashbuckle.AspNetCore" Version="5.0.0-rc4" />
        <PackageReference Include="Swashbuckle.AspNetCore.Annotations" Version="5.0.0-rc4" />
        <PackageReference Include="Swashbuckle.AspNetCore.Filters" Version="5.0.0-rc8" />
    </ItemGroup>

    ...

</Project>

[jkotalik]

The part in your csproj concerns:

    <ItemGroup>
        <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.0.0" />
        <PackageReference Include="Microsoft.AspNetCore.Authorization" Version="3.0.0" />
        **<PackageReference Include="Microsoft.AspNetCore.Hosting" Version="2.2.7" />**
        **<PackageReference Include="Microsoft.AspNetCore.Server.IIS" Version="2.2.6" />**
        <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="3.0.1" />
        <PackageReference Include="Microsoft.Extensions.Hosting" Version="3.0.1" />
        <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="3.0.0" />
        <PackageReference Include="Swashbuckle.AspNetCore" Version="5.0.0-rc4" />
        <PackageReference Include="Swashbuckle.AspNetCore.Annotations" Version="5.0.0-rc4" />
        <PackageReference Include="Swashbuckle.AspNetCore.Filters" Version="5.0.0-rc8" />
    </ItemGroup>

You should not be mixing 3.0 and 2.2 dependencies like that. I think you can honestly remove the references from your project, they should be implicitly brought in by referencing the websdk. See https://docs.microsoft.com/en-us/aspnet/core/migration/22-to-30?view=aspnetcore-3.1&tabs=visual-studio#framework-reference for more info. I wouldn't be surprised if this is causing some of these odd behaviors you are seeing.

Besides that, you can check the windows event log for more information about the execptions you are seeing.

[msftbot[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.

[Siliconrob]

I am having this same issue. It is related to the Visual Studio 16.4 update, please do not close or mark this issue stale

[jkotalik]

@Siliconrob could you post what you have in your csproj? From my original analysis, this issue only affected application that were crossing dependencies between 2.2 and 3.0, which isn't a supported scenario. If that isn't the case, I'd like to start investigating this asap as I've seen two issues about it.

[Siliconrob]

```

netcoreapp3.0

all
runtime; build; native; contentfiles; analyzers; buildtransitive




















```

[jkotalik]

Yeah, it looks like you are mixing versions between 2.2 and 3.0. Can you remove all Microsoft.AspNetCore packages with 2.2 in them? Those should all be implicitly added, I'd imagine there is something going wrong there.

[jkotalik]

See https://docs.microsoft.com/en-us/aspnet/core/migration/22-to-30?view=aspnetcore-3.1&tabs=visual-studio as well.

[Siliconrob]

```

netcoreapp3.0

all
runtime; build; native; contentfiles; analyzers; buildtransitive




















```

Removed the 2.2 references and it will launch with OutOfProcess set, but not with InProcess specified

[jkotalik]

Can you check the windows event log to see what error is being thrown? Also, can you see if you are able to debug break at the start of program main. The original issue mentioned they were able to get to ConfigureServices, but not Configure.

[Siliconrob]

Application '/LM/W3SVC/21/ROOT' with physical root '<my_project_here>' hit unexpected managed background thread exit, exit code = '0'. Please check the stderr logs for more information.

Setting the log output there is nothing in it. The file is empty. .\logs\stdout

Example
stdoutLogFile=".\logs\stdout" stdoutLogEnabled="true" hostingModel="InProcess"

Doesn't reach either. It exists immediately. I mean this is more of annoying issue now instead of blocking as am able to set to OutOfProcess and everything works nicely again

[jkotalik]

For the file .\logs\stdout, you'd need to set stdoutLogEnabled="true". Otherwise, I'm definitely confused. The exit code was 0, meaning the app exited "gracefully". I don't think it's an issue with VS 16.4 itself; it seems like an issue with .NET Core 3.1 itself.

Can you show me what you are doing in program main, or potentially a repro app would be useful

[Siliconrob]

Going to need to strip this project down to something for a reproducible app, but I did set the stdoutLogEnabled="true"

This was file generated stdout_20191209232503_14508.log and it's empty so nothing to attach

[jkotalik]

Cool, please let me know when you have a repro.

[patilkunal50]

Change your module name
from AspNetCoreModuleV2
to AspNetCoreModule in your csproj. It will resolve your issue.

[Siliconrob]

Change your module name
from AspNetCoreModuleV2
to AspNetCoreModule in your csproj. It will resolve your issue.

1. Revert to the earlier version of .NET Core?
2. This doesn't work anyway

[jkotalik]

@Siliconrob any luck getting a repro app?

[patilkunal50]

Try changing it in your web.config as well!

<system.webServer> <handlers> <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" /> </handlers> <aspNetCore processPath="dotnet" arguments=".\FreConsultants.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="InProcess" /> </system.webServer>

[Siliconrob]

@Siliconrob any luck getting a repro app?

Setting a deadline of 24 hours to get you that starting now. OK?

[saysmaster]

The part in your csproj concerns:

  <ItemGroup>
      <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.0.0" />
      <PackageReference Include="Microsoft.AspNetCore.Authorization" Version="3.0.0" />
      **<PackageReference Include="Microsoft.AspNetCore.Hosting" Version="2.2.7" />**
      **<PackageReference Include="Microsoft.AspNetCore.Server.IIS" Version="2.2.6" />**
      <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="3.0.1" />
      <PackageReference Include="Microsoft.Extensions.Hosting" Version="3.0.1" />
      <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="3.0.0" />
      <PackageReference Include="Swashbuckle.AspNetCore" Version="5.0.0-rc4" />
      <PackageReference Include="Swashbuckle.AspNetCore.Annotations" Version="5.0.0-rc4" />
      <PackageReference Include="Swashbuckle.AspNetCore.Filters" Version="5.0.0-rc8" />
  </ItemGroup>

You should not be mixing 3.0 and 2.2 dependencies like that. I think you can honestly remove the references from your project, they should be implicitly brought in by referencing the websdk. See https://docs.microsoft.com/en-us/aspnet/core/migration/22-to-30?view=aspnetcore-3.1&tabs=visual-studio#framework-reference for more info. I wouldn't be surprised if this is causing some of these odd behaviors you are seeing.

Besides that, you can check the windows event log for more information about the execptions you are seeing.

Hello, I'm sorry for the late response. I was extremely busy at work and did not have time to answer. I will try as soon as possible to fix the package references and see if it is the source of the problem.

As I said, other 3.0.1 apps were running successfully.

The thing is, in the production environment there is only asp.net core runtime and hosting bundle for 3.0.1 installed, so I hope removing the package references doesn't break anything, since I will be building with 3.1 SDK.

[jkotalik]

As I said, other 3.0.1 apps were running successfully.

For sure it's interesting; if you are able to get me a repro app, I can start to try to identify what the underlying problem for why 3.0.1 worked and 3.1 didn't.

[Siliconrob]

@Siliconrob any luck getting a repro app?

Setting a deadline of 24 hours to get you that starting now. OK?

I'm not going to make it other things have come up. I will get to this when I can (sometime later this week). Sorry, there is a lot of proprietary code that must be removed and anonymized prior to me giving you a repro app

[anurse]

Triage: We're pretty stuck here without a repro :(. Can anyone on this thread provide something? We aren't able to repro this.

[Siliconrob]

Triage: We're pretty stuck here without a repro :(. Can anyone on this thread provide something? We aren't able to repro this.

Yeah I know, let you down on this. Will work on it some more when I have a chance

[patilkunal50]

Remove all the package reference versions below 3.0 from the project and clean and re-build the solution. This worked for me!

[anurse]

Remove all the package reference versions below 3.0 from the project and clean and re-build the solution. This worked for me!

This is what I would expect to work. Using 2.x versions of Microsoft.Extensions.* and Microsoft.AspNetCore.* packages in an ASP.NET Core 3.0/3.1 project is *not supported.

Yeah I know, let you down on this. Will work on it some more when I have a chance

No worries! We'll be here when you get something together. We're very aware that other things can get in the way of setting up a repro, it's totally fine!

[patilkunal50]

Remove all the package reference versions below 3.0 from the project and clean and re-build the solution. This worked for me!

This is what I would expect to work. Using 2.x versions of Microsoft.Extensions.* and Microsoft.AspNetCore.* packages in an ASP.NET Core 3.0/3.1 project is *_not supported_.

Yeah I know, let you down on this. Will work on it some more when I have a chance

No worries! We'll be here when you get something together. We're very aware that other things can get in the way of setting up a repro, it's totally fine!

Yes, exactly my point package references below 3.0 are not supported. Just remove those references and the issue will be resolved.

[Siliconrob]

Remove all the package reference versions below 3.0 from the project and clean and re-build the solution. This worked for me!

This is what I would expect to work. Using 2.x versions of Microsoft.Extensions.* and Microsoft.AspNetCore.* packages in an ASP.NET Core 3.0/3.1 project is *_not supported_.

Yeah I know, let you down on this. Will work on it some more when I have a chance

No worries! We'll be here when you get something together. We're very aware that other things can get in the way of setting up a repro, it's totally fine!

Yes, exactly my point package references below 3.0 are not supported. Just remove those references and the issue will be resolved.

I know your trying to help, but I did that and it did not resolve the issue as I stated in an earlier response. It could be that some of the Dependency injected libraries are causing pre 3.0 references and I need to go through them all, but at the project level there are no 3.0 refs and it breaks. This also started happening in another 3.0 app which we have also converted to OutOfProcess for now as well

[anurse]

@Siliconrob can you post your full csproj after you removed the references? I see a number of 2.x references in your earlier csproj. It's possible one is coming in transitively from a third-party dependency, which would also cause this problem.

[saysmaster]

Hello. Today I had the time to look at the project again.

Indeed, I can confirm that, in my case the problem was those two Nuget packages, as @jkotalik mentioned:

<PackageReference Include="Microsoft.AspNetCore.Hosting" Version="2.2.7" />
<PackageReference Include="Microsoft.AspNetCore.Server.IIS" Version="2.2.6" />

Removing them I could start the 3.0 project with the 3.1 SDK and InProcess.
I don't even remember how did those two packages end up there in my csproj. Other 3.0 projects in my solution don't reference those packages.

Thank you @jkotalik

[HarelM]

I'm experiencing this issue, or a similar one...
The following project started failing after I upgraded to vs 16.4 from 16.3.
https://github.com/IsraelHikingMap/Site/
The exception is the following:

Object reference not set to an instance of an object.
   at Microsoft.AspNetCore.Hosting.WebHostBuilderIISExtensions.<>c__DisplayClass0_0.<UseIIS>b__2(IISServerOptions options)
   at Microsoft.Extensions.Options.ConfigureNamedOptions`1.Configure(String name, TOptions options)
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at Microsoft.Extensions.Options.OptionsManager`1.<>c__DisplayClass5_0.<Get>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()

When changing to OutOfProcess it works well and doesn't crash.
I suspect it has something to do with the fact that I have a web.config file in my project (IsraelHiking,Web) but it did work in 16.3 and now it crashes...
Any help would be appreciated.

[jkotalik]

@HarelM apologies I should have informed this closed issue as well. We found a regression in the 3.1.0 and 3.0.1 SDK. See https://github.com/aspnet/Announcements/issues/398

[HarelM]

@jkotalik thanks for the link but the workaround is not clear enough for me to understand what I should do in order to work around the issue. Furthermore ever since I upgraded my server to .net core 3.0 I keep hitting random json deserialization problems in multi-threading that I currently believe the only solution is to downgrade back to 2.x. Assuming the root cause is this upgrade and not another package I upgraded I'll report it here.
Also the change from 16.3 to 16.4 changed the deps.json file when publishing.
All in all, a very poor experience of things breaking lately when using latest versions... (I spent an hour yesterday on this trying to get the environment up and running, which it did perfectly before the VS upgrade...)