Aspnetcore: Installing ASP.NET Core Hosting Bundle 3.1.0 crashes 32 bit application pools

[bshade7]

The Problem

I have an existing IIS Web Server hosting several ASP.NET (Non-Core) web applications that are utilizing the "Enable 32 bit Application" setting on their respective application pools.

Upon installation of the latest ASP.NET Core Hosting Bundle (3.1.0), all application pools for my existing sites crash and remain stopped after crashing 5 times quickly.

This seems to only impact the sites that are using the 32bit application setting as I have several others that are compiled as "Any CPU" and are using the 64 bit application pool type without issue.

I also have several new .NET Core web applications that are also working fine in 64bit mode (since that is how they were compiled in the first place). This is also my reason for needing the hosting bundle installed in the first place.

The error seen in the Windows Event Log is as follows each time the application pools crash (every existing web application throws the same error when trying to start).

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: aspnetcorev2.dll, version: 13.1.19320.0, time stamp: 0x5dcfdb7b
Exception code: 0xc000001d
Fault offset: 0x00026ed9
Faulting process id: 0x106c
Faulting application start time: 0x01d5b9e26b09b5f7
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Program Files (x86)\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll
Report Id: a8bff24d-25d5-11ea-80c8-005056957661
Faulting package full name: 
Faulting package-relative application ID:

Further technical details

ASP.NET Core 3.1.0

Output of dotnet --info

Host (useful for support):
  Version: 3.1.0
  Commit:  157910edee

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Installed items:

Please let me know what else I can provide to get assistance with this issue. Thank you!

[jkotalik]

So I looked up the error you are hitting (0xc000001d)

Instruction An attempt was made to execute an illegal instruction.

This looks like the 32 bit dll is potentially corrupted in some way. Can you see what happens if you uninstall the 3.1.0 hosting bundle and install the 3.0.1 bundle? Or can you see what happens if you uninstall and reinstall 3.0.1?

[jkotalik]

cc @joeloff

[bshade7]

@jkotalik I should be able to give this a shot within the next couple days, I will update the results of trying the 3.0.1 hosting bundle instead. Thanks for the response.

[bshade7]

I was able to give this a shot today and the results with 3.0.1 are the exact same as what I was seeing with 3.1.0. The following error messages were written to the windows event application log:

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: aspnetcorev2.dll, version: 13.0.19309.1, time stamp: 0x5dc0c3ae
Exception code: 0xc000001d
Fault offset: 0x00026ed9
Faulting process id: 0xdd0
Faulting application start time: 0x01d5bebd07e15d31
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Program Files (x86)\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll
Report Id: 4599d5b6-2ab0-11ea-80d4-0050569519d2
Faulting package full name: 
Faulting package-relative application ID:

As you can see, the dll version changed, but apart from that the results are the same. Is there something I am missing to have the non-aspnetcore apps configured? I have tried adding remove statements in the web.config, but nothing seems to get me anything different.

Let me know what else you need from me to help get to the root of what is going on here. Thank you.

[jkotalik]

Very weird. I've tried to repro this locally with the 3.0.1 installation of IIS and I couldn't repro it. The only thing that I can think of is the version of Windows you have.

For my sanity, can you check that the bitness of the aspnetcorev2.dll is 32 bit? You can use something like dumpbin to find the bitness.

[lousaibiao]

any update? facing the same issue

[jkotalik]

@lousaibiao can you try doing what I mentioned in my previous comment and check the bitness of the aspnetcorev2.dll? We are still trying to gather more information.

[lousaibiao]

where can i find dumpbin?

[lousaibiao]

@jkotalik

[jkotalik]

https://docs.microsoft.com/en-us/cpp/build/reference/dumpbin-command-line?view=vs-2019

If you open Visual Studio 2019 Developer Command Prompt, you should be able to type:
dumpbin /headers "C:\Program Files (x86)\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll". It should then list the bitness of the dll (either x64 or x86)

[lousaibiao]

@jkotalik wait minutes, I am trying

[lousaibiao]

@jkotalik it says dumpbin is missing, how can i fix this.

[lousaibiao]

@jkotalik I have two windows server r2 standard machine. it works in one but crash in another. Also, the aspnetcorev2.dll file in two machine seems exactly the same. Did i missing something in the wrong machine?

[bshade7]

@jkotalik Here are the results of the two dlls i have from 3.1.0, since I reverted back to that version to keep my .net core applications working properly (since they were made for 3.1.0). Bitness seems to be correct for both dlls present on the machine.

[mvarblow]

We're seeing the same thing. We installed the ASP.NET Core 3.1 Hosting Bundle on a server so we could deploy a new .NET Core 3.1 application. This broke all our ASP.NET Core 2.2 (.NET Framework 4.7.2) applications running on the server in 32-bit application pools. Same error codes / messages as described by bshade7. We're running on Windows Server 2012 R2.

[lousaibiao]

This not happen in every machine,i think it
may be caused by some missing Windows updates,because my customer trun off auto update. Finally,i installed hosting bundle with version 3.0.0 so it would not be so much changes to my code.

[jkotalik]

Alright, I'm going to try to repro this on Windows Server 2012 R2 as that seems to be the consistent thing here.

[jkotalik]

I tried to repro this without any luck. Here is what I did to let everyone compare:

• I created a Windows Server 2012 R2 Datacenter skew in Azure
• Once created, I downloaded the 3.1 hosting bundle and ran it on the server. This is the same version as listed by @bshade7 at the beginning
• I published a 32 bit aspnetcore website and copied it to the inetpub folder
• I hit the site on localhost to check if there was an exception.

And I was successfully able to hit the website from both inprocess and out of process.

At this point, I'm not really sure where to go. Here is what I have for versions

• Windows Version 6.3.9600
• w3wp.exe: 8.5.9600.16384 (same as original repro)

[bshade7]

@jkotalik I also have the same Windows Version, however my edition is different. I am using Windows Server 2012 R2 Standard (Still build 6.3.9600 though). I am not sure if that makes any difference or not.

This machine is also connected to a corporate domain (if that matters), and similar to what @lousaibiao mentioned, they hold updates and distribute them quarterly, so the machine may not have the "latest" updates.

What else can we try to capture to figure out what is going on here?

[bshade7]

@jkotalik I am in the process of trying to capture a debug dump of the w3wp process using Debug Diagnostics 2 Update 2 for what I am experiencing. If I can grab one of these, would you like me to send it to you? If so, where should I put it? Thank you.

[bshade7]

@jkotalik I have captured several dumps of the process crashing. I have attached the analysis result from the crash.

Here is the error from the top of the analysis:

In w3wp__SVAppPool__PID__15304__Date__01_07_2020__Time_11_56_10AM__795__First 
Chance Illegal Instruction.dmp the assembly instruction at aspnetcorev2!_ftoui3+9 
in C:\Program Files (x86)\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll from 
Microsoft has caused an unknown exception (0xc000001d) on thread 3

Here is the stack trace from Thread 3 that caused the exception:

Entry point | ntdll!RtlRegisterThreadWithCsrss+50
Create time | 1/7/2020 11:56:00 AM
Time spent in user mode | 0 Days 00:00:00.000
Time spent in kernel mode | 0 Days 00:00:00.046


aspnetcorev2!_ftoui3+9
aspnetcorev2!RegisterModule+178
iiscore!W3_SERVER::LoadModulesFromConfig+34e
iiscore!W3_SERVER::InitializeGlobalModules+28
iiscore!IISCORE_PROTOCOL_MANAGER::InitializeGlobals+200
iiscore!IISCORE_PROTOCOL_MANAGER::PreloadApplication+3a
w3wphost!W3WP_HOST::ProcessHttpPreloadApplications+64
w3wphost!W3WP_HOST::ProcessPreloadApplications+6d
w3wphost!WP_IPM::AcceptMessage+1b5
iisutil!IPM_MESSAGE_PIPE::MessagePipeCompletion+4ca
ntdll!RtlDeregisterWaitEx+f2
ntdll!WinSqmAddToStreamEx+472
ntdll!WinSqmAddToStreamEx+3dc
ntdll!WinSqmAddToStreamEx+2d6
kernel32!BaseThreadInitThunk+24
ntdll!RtlInitializeExceptionChain+8f
ntdll!RtlInitializeExceptionChain+5a

And finally, here is the Exception Information from the bottom of the analysis:

Module Information
--
Image Name: | C:\Program Files (x86)\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll | Symbol Type: | PDB
Base address: | 0x00905a4d | Time Stamp: | Sat Nov 16 05:20:27 2019
Checksum: | 0x00000000 | Comments: |  
COM DLL: | False | Company Name: | Microsoft
ISAPIExtension: | False | File Description: | IIS ASP.NET Core Module V2. Commit: 2b7e994b8a304700a09617ffc5052f0d943bbcba
ISAPIFilter: | False | File Version: | 13.1.19320.0
Managed DLL: | False | Internal Name: | aspnetcore
VB DLL: | False | Legal Copyright: | Copyright (C) Microsoft Corporation
Loaded Image Name: | aspnetcorev2.dll | Legal Trademarks: |  
Mapped Image Name: |   | Original filename: | aspnetcore.dll
Module name: | aspnetcorev2 | Private Build: |  
Single Threaded: | False | Product Name: | ASP.NET Core Module
Module Size: | 264 KBytes | Product Version: | 13.1.19320.0
Symbol File Name: | c:\symbols\aspnetcorev2.pdb\0E1025C92EF14F4DB3D64699E6CC1D8D1\aspnetcorev2.pdb | Special Build: | &

Please let me know if you would like the full dump from the crash that was analyzed to get this. Thank you.
CrashAnalysis.zip

[jkotalik]

I really am stumped here. I don't believe there should be any difference in the way I verified ANCM and what you have. I can maybe try getting the non datacenter skew, but that's about it. The only thing that looks interesting is in your dump is:

Loaded Image Name: | aspnetcorev2.dll | Legal Trademarks: |  
Mapped Image Name: |   | Original filename: | aspnetcore.dll

I don't know why aspnetcore.dll is the mapped image there rather than aspnetcorev2.dll.

I think I need another set of eyes. @anurse / @Tratcher ?

[mvarblow]

Are others having this problem when running applications built using the ASP.NET Core 2.2 libraries and targeting x86 .NET Framework (rather than .NET Core)? That's our situation.

We can reproduce the problem consistently on one server that's running Windows Server 2012 R2 Standard (6.3.9600) with all available updates applied (last installed 12/11/2019). We have another server running Windows Server 2012 R2 Datacenter (6.3.9600). This second server also has all available updates applied (last installed 12/11/2019) but we have not been able to reproduce the problem on the second server.

The message we see in the event log looks the same as others have reported above:

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: aspnetcorev2.dll, version: 13.1.19320.0, time stamp: 0x5dcfdb7b
Exception code: 0xc000001d
Fault offset: 0x00026ed9
Faulting process id: 0x8dc4
Faulting application start time: 0x01d5c568e2eb3547
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Program Files (x86)\IIS\Asp.Net Core Module\V2aspnetcorev2.dll

[bshade7]

@mvarblow At least in my situation, the apps that are crashing were built well before ASP.NET Core was around to do so. However, they are absolutely targeting the x86 .NET Framework (NOT .NET Core). So perhaps you are onto something there. Maybe there is an issue when targeting the x86 (non Core) framework and utilizing ANCM to host the app with IIS? Just spitballing here trying to help out the devs.

Thoughts?

[jkotalik]

It seems like no matter what, I should verify the Standard SKU instead of the Datacenter. I'll start spinning up a VM for that.

I don't think .NET Framework vs .NET Core should make a difference inside of ANCM. No managed code has been loaded by that point.

[jkotalik]

So I tried again today on a 2012 R2 Standard and I wasn't able to repro again.

One idea we came up with; can you try downloading the latest Visual C++ redistributable?
See https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads.

[bshade7]

@jkotalik I actually had already installed this after reading about some other issues people have reported. Here is the versions I have installed on that machine.

[lousaibiao]

I also installed this Visual C++ redistribution already

[anurse]

I have an existing IIS Web Server hosting several ASP.NET (Non-Core) web applications that are utilizing the "Enable 32 bit Application" setting on their respective application pools.

@bshade7 can you clarify something here? Are the app pools that are crashing hosting System.Web-based applications? You mentioned "non-Core" so I want to make sure I'm understanding the situation properly.

This broke all our ASP.NET Core 2.2 (.NET Framework 4.7.2) applications running on the server in 32-bit application pools.

@mvarblow In your case, it's crashing app pools running ASP.NET Core, but using .NET Framework, correct?

---

And to be clear, do any of you have .NET Core applications using the same app pool as applications using .NET Framework (System.Web, _or_ ASP.NET Core)? I believe that is not supported (@shirhatti @jkotalik @Tratcher ?).

[bshade7]

@anurse Yes, that is correct, the app pools that are crashing are hosting System.Web applications. Also, I am not sharing an app pool for any application actually. So in my case, I am definitely not sharing an app pool with the .NET Core applications. Each application has its own app pool. Please let me know if you have any more questions. Thanks!

[mvarblow]

@anurse That is correct. The crashing application pool contains an IIS application running ASP.NET Core using .NET Framework 4.7.2. This same application pool also hosts another IIS application running on System.Web and .NET Framework 4.7.2.

[mvarblow]

@jkotalik, installing the latest C++ redistributable (and rebooting) did not resolve the problem.

[artemsv]

In our case the application that is crashed is 32-bit ISAPI Extension written in C++ decade ago. AppPool configured as "No Managed Code" and Integrated Pipeline Mode. We have installed on the server with tens of that old applications ASP Net Core Hosting Bundle 3.1.0 and now each access from browser to that ISAPI dll causes the same error in aspnetcorev2.dll.

Maybe this fact can help you to look at other points...

[jkotalik]

In our case the application that is crashed is 32-bit ISAPI Extension written in C++ decade ago. AppPool configured as "No Managed Code" and Integrated Pipeline Mode. We have installed on the server with tens of that old applications ASP Net Core Hosting Bundle 3.1.0 and now each access from browser to that ISAPI dll causes the same error in aspnetcorev2.dll.

So it sounds like the pure existence of the dll inside of the applicationhost.config causes this error, which isn't surprising based on the error that is happening. I believe no matter what, IIS calls LoadModule on all dlls, and it's failing there.

Here is all of the information I know so far, feel free to correct anything if there is differences in your current scenario:

• Repros only on Windows Server 2012 R2 Standard with 32 bit application pools
• Repros with 3.0.1 and 3.1.0. One thing I would like to check is if this repros with 3.0.0 as well. Can one of the people hitting this issue try using 3.0.0 from here: https://dotnet.microsoft.com/download/dotnet-core/3.0. Also trying 3.0.2/3.1.1 which released today could be something to try as well.
• Installing the latest C++ redistributable didn't help.
• Seems to be an error purely with native code and loading the dll.
• The error _ftoui3 seems to be a function that is part of the VC++ sdk.

One thing I may try is building a custom build of ANCMv2 that targets a different VC++ SDK. We update this in 2.2 to 3.0: https://github.com/dotnet/aspnetcore/blame/master/src/Servers/IIS/AspNetCoreModuleV2/AspNetCore/AspNetCore.vcxproj#L57. I'd be curious if that affected this in any way.

Again, apologies to anyone hitting this issue. I'm trying my best to get to the bottom of this. Thank you for your patience.

[lousaibiao]

In my machine, failed with 3.0.1 and 3.1.0, but works with 3.0.0

[anurse]

This looks a lot like a similar root cause as https://github.com/dotnet/wpf/issues/2259 . I'm looking through that thread now, but it looks like the problem is due to a bug in the VS 16.3 C++ tools (which we use for building ANCM). It's fixed in 16.4 so we'll plan to make a change to update to that version which should hopefully resolve this issue.

In the meantime I think there's a workaround you all can apply to fix this temporarily, which I'm investigating now and will post back with more information. Unfortunately the workaround doesn't work in ANCM's case.

[anurse]

Btw, it looks like this bug it conditional on certain kinds of CPU (since the bug is faulty CPU detection) so that's why it's been so hard to reproduce!

[EmperorZeppo]

I don't have a lot to add here. Basically I'm seeing the same issue (.Net Framework 4 app, AppPool 32-Bit True, .Net Core 3.1 Host Bundle, Exception code: 0xc000001d). Just adding to the list of folks impacted. Looking for any updates.

[jkotalik]

@MattGal / @mmitche / @dotnet/aspdoi what steps do we need to take to move to 16.4 in aspnetcore? We are currently using 16.3, but due to a bug in the 16.3 C++ tools, we need to move to 16.4 (or figure out how to isolate using the C++ tools from 16.4, but I'd imaging moving to 16.4 may be easier).

[MattGal]

@MattGal / @mmitche / @dotnet/aspdoi what steps do we need to take to move to 16.4 in aspnetcore? We are currently using 16.3, but due to a bug in the 16.3 C++ tools, we need to move to 16.4 (or figure out how to isolate using the C++ tools from 16.4, but I'd imaging moving to 16.4 may be easier).

The BuildPool machines with VS 2019 will naturally move forward to the latest public minor release when updated; @v-joshli I believe updates them and may be able to comment.

[sanmolhec]

Hello.

We've had a similar experience in our organization.

Yesterday, Tuesday, we installed NET Core Hosting 3.1 on a w2k12 Server and all the pools that were forced to enable 32-bit applications to true do not start.

We have uninstalled Core 3.1 hosting but the problem remains.

On the production server we are going back to a previous restore point, but we have left the problem on the development server to try and fix it (and then move the solution to production).

Any help you can give us is welcome.

Thank you very much.

Best regards, Héctor.

[artemsv]

Our admins says that they changed EC2 instance type from r5.xlarge to r5a.xlarge (with AMD CPU) and the issue disappeared!

[asianx]

Hello,
Was facing a similar issue as others here. After installing ASP.NET Core Hosting Bundle 3.1.1, all 32 bit application pools were stopping automatically upon trying to access the site/application through web browser.

Based on the event logs was getting the following:
Faulting module path: C:\Program Files (x86)\IIS\Asp.Net Core Module\V2aspnetcorev2.dll

After uninstalling the 3.1.1 bundle, was still getting the same error so I decided to overwrite the dlls in the above path with the last stable version we had. That looks to have resolved the issue for us.

[joeloff]

I'll need to double check, but copying the old file is likely required because ANCM upgrades in place. So if you have multiple ANCM installs and then remove the one that installed the highest version of the DLL, but an older MSI holds a reference to the DLL, the ref count would simply be adjusted and the newer copy would remain. Your option is to either replace the file manually or to delete it and then repair the last known version of ANCM that was installed to place the old copy of the file back on disk.

[jkotalik]

@joeloff is correct. Copying the old files is required. Another option is to do the following:
Install the 3.0.0 hosting bundle with the flags: OPT_NO_RUNTIME, OPT_NO_SHAREDFX
Install the 3.1.1 hosting bundle with the flag: OPT_NO_ANCM

usage:

dotnet-hosting.exe OPT_NO_RUNTIME=1 OPT_NO_SHAREDFX=1

I'm discussing with our build team ways to get a fix out as soon as possible. Thanks you all for your patience.

For documentation on the hosting bundle, see https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/?view=aspnetcore-3.1#install-the-hosting-bundle.

[anurse]

Our admins says that they changed EC2 instance type from r5.xlarge to r5a.xlarge (with AMD CPU) and the issue disappeared!

Yep, that's what I would expect. The issue is specific to particular CPU models 😲! We're working on getting a fix together now.

[niti01]

Hi,

We had the asp.net core 2.2 hosting module installed and after installing 3.1.0 we experienced this issue with all existing 32-bit app pools crashing.

We uninstalled 3.1.0 and tried @jkotalik 's solution with installing 3.0.0 and 3.1.1 bundles with the specified flags. However, we still do not get the file: C:\Program Files (x86)\IIS\Asp.Net Core Module\V2aspnetcorev2.dll which we expected the 3.0.0 bundle to install. According to the attached 3.0.0 installation log, the installer seems to recognize that the file is absent but it is not installed anyway.

dd_dd_DotNetCoreWinSvrHosting__20200205164010.log

And we do not have the aspnetcorev2.dll from 3.0.0 as we had not installed this version previously.

[joeloff]

Based on the log, it detected a newer version for ANCM to be already present on the machine and marked the MSI as obsolete

[0ACC:0744][2020-02-05T16:40:10]i103: Detected related package: {C2109FCE-737E-479E-98EC-039A39840AB9}, scope: PerMachine, version: 13.1.19350.0, language: 0 operation: Downgrade
[0ACC:0744][2020-02-05T16:40:10]i201: Planned package: AspNetCoreModuleV2_x64, state: Obsolete, default requested: None, ba requested: None, execute: None, rollback: None, cache: No, uncache: No, dependency: None

[niti01]

Based on the log, it detected a newer version for ANCM to be already present on the machine and marked the MSI as obsolete

Ok. I also tried to rename/delete the x64-version of aspnetcorev2.dll but still the 3.0.0 installation package did not install aspnetcorev2.dll (netiher x32 nor x64). I guess it somehow still considers it already installed. I manually copied v2.2 of the dll to the V2 folder. And now the existing 32-bit application does not crash at least. But we are eagerly awaiting the fix...

[victormss]

We also have this problem. It only happens in virtual machines with Windows 2012, in which 2016, 2019 and physical servers with 2012 does not happen. All the hosts are Windows 2019.
The hosts and physical servers have different cpus, but they are all Intel.
The installed version is 3.1.1

[codingforfoodllc]

I don't really have a lot to add other than we we are also having this issue. 32 bit applications pools are crashes when we installed core hosting 3.1.0.

[jkotalik]

This will be fixed in 3.1.3.

[codingforfoodllc]

Awesome! Is there a timeline for 3.1.3 release?

[anurse]

We're releasing the 3.1.2 release today, and generally aim for monthly (though there's no guarantee), so we expect early March for 3.1.3. In general though, we don't have a committed timeline for patch releases. Hope that helps!

[RobPierce]

This will be fixed in 3.1.3.

Is this version 3.1.3 of the hosting bundle, or is it the sdk that is the ultimate problem?

[jkotalik]

The hosting bundle,

[jkotalik]

3.1.3 is now available: https://dotnet.microsoft.com/download/dotnet-core/3.1. This issue should now be resolved in 3.1.3