Aspnetcore: In-Process hosting deployment to Azure from command line

[cdwaddell]

When I deploy my .Net core application to Azure using the command line tools in VSTS:

dotnet publish --configuration Release --output $(Build.ArtifactStagingDirectory)

I receive this error when I webdeploy and launch my app:

HTTP Error 500.0 - ANCM In-Process Handler Load Failure

When doing the same deployment using Visual Studio publish tools, the application loads as expected.
```

MSDeploy
AzureWebSite
Release
Any CPU
False
False
True
[My URL]
[AppName]

True
WMSVC
True
$[AppName]
<_SavePWD>True
<_DestinationType>AzureWebSite
False
netcoreapp2.2
34fc2225-66cb-43ea-bce0-8793e51fed2e
false

When using Kudu to inspect the application folder, they contain the same files after each of the deployments. I also have the following .Net Core Extension installed:

> AspNetCoreRuntime.2.2.x64 

The web.config has the same declarations after both deployments:

```
When my application is deployed after disabling InProcess the deployment works from the command line and from Visual Studio. I have tried various different command line switches, including setting the runtime and the framework versions, but I continue to get the aforementioned error.

Is this process supported?

[jkotalik]

This should be supported. Can you check a few things:

• Any event logs coming from ANCM?
• You mentioned you tried different framework versions, does that mean you tried standalone/self-contained deployment? If not, try that.
• Can you see if the file [MyApp].deps.json contains the file "aspnetcorev2_inprocess.dll" under runtime identifiers.

[cdwaddell]

Any event logs coming from ANCM?

I do not see any event logs from my other attempts, although I may be looking in the wrong place for these. I enabled Diagnostic Logs and do not see anything. The event viewer in Kudu is apparently no longer there, its old location pointed me to the Diagnostic Logs.

You mentioned you tried different framework versions, does that mean you tried standalone/self-contained deployment? If not, try that.

Yes, I tried both standalone and self-contained deployments. For standalone, I tried win-x64 and win10-x64 runtimes. I only tried with netcoreapp2.2. Both standalone and self-contained work with Visual Studio "Publish", but neither work with the dotnet publish command line interface. They both give me the same error "HTTP Error 500.0 - ANCM In-Process Handler Load Failure"

Can you see if the file [MyApp].deps.json contains the file "aspnetcorev2_inprocess.dll" under runtime identifiers.

When using both the command line interface and Visual Studio to publish the app, "aspnetcorev2_inprocess.dll" is not in my [MyApp].deps.json file. I did a diff of this file between VS Publish method and dotnet publish method and they are practically the same.

[cdwaddell]

To make sure the issue wasn't on the VSTS side, I ran the build locally and uploaded the files via ftp.
dotnet publish -c Release --self-contained -r win10-x64
Everything builds and deploys. The application that built runs locally via the exe, but I get the same error when hosted in Azure:

HTTP Error 500.0 - ANCM In-Process Handler Load Failure

To fix, I just load Visual Studio and "Publish" the exact same application to the exact same Azure Web App. What is required via the command line to get an In-Process application running in Azure?

[jkotalik]

Can you try enabling debug logs and post the output here? See: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/troubleshoot?view=aspnetcore-2.2#enable-the-aspnet-core-module-debug-log.

[cdwaddell]

Ok, I now have logs. Here is my log from dotnet publish. It has an error and is running with "aspnetcorev2.dll".

[aspnetcorev2.dll] Initializing logs for 'D:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 33752.. File Version: 12.2.18296.0. Description: IIS ASP.NET Core Module V2. Commit: 61f1a70784dc0a32cf98f8ddd169c0293b0390ab.
[aspnetcorev2.dll] Resolving hostfxr parameters for application: '.\[MyApp].exe' arguments: '' path: 'D:\home\site\wwwroot\'
[aspnetcorev2.dll] Known dotnet.exe location: ''
[aspnetcorev2.dll] Process path '.\[MyApp].exe' is not dotnet, treating application as standalone or portable with bootstrapper
[aspnetcorev2.dll] Checking application.dll at 'D:\home\site\wwwroot\.\[MyApp].dll'
[aspnetcorev2.dll] Checking hostfxr.dll at 'D:\home\site\wwwroot\.\hostfxr.dll'
[aspnetcorev2.dll] hostfxr.dll found app local at 'D:\home\site\wwwroot\.\hostfxr.dll', treating application as standalone
[aspnetcorev2.dll] Parsed hostfxr options: dotnet location: '' hostfxr path: 'D:\home\site\wwwroot\.\hostfxr.dll' arguments:
[aspnetcorev2.dll] Argument[0] = 'D:\home\site\wwwroot\.\[MyApp].exe'
[aspnetcorev2.dll] Event Log: 'Invoking hostfxr to find the inprocess request handler failed without finding any native dependencies. This most likely means the app is misconfigured, please check the versions of Microsoft.NetCore.App and Microsoft.AspNetCore.App that are targeted by the application and are installed on the machine.' 
End Event Log Message.
[aspnetcorev2.dll] Failed HRESULT returned: 0x8000ffff at c:\b\w\e37dd45d8cd1eaf4\modules\iisintegration\src\aspnetcoremodulev2\aspnetcore\handlerresolver.cpp:80 
[aspnetcorev2.dll] Event Log: 'Could not find inprocess request handler. Captured output from invoking hostfxr: Invalid runtimeconfig.json [D:\home\site\wwwroot\[MyApp].runtimeconfig.json] [D:\home\site\wwwroot\[MyApp].runtimeconfig.dev.json]
' 
End Event Log Message.
[aspnetcorev2.dll] Failed HRESULT returned: 0x8000ffff at c:\b\w\e37dd45d8cd1eaf4\modules\iisintegration\src\aspnetcoremodulev2\aspnetcore\handlerresolver.cpp:153 
[aspnetcorev2.dll] Failed HRESULT returned: 0x8000ffff at c:\b\w\e37dd45d8cd1eaf4\modules\iisintegration\src\aspnetcoremodulev2\aspnetcore\applicationinfo.cpp:136 
[aspnetcorev2.dll] Failed HRESULT returned: 0x8000ffff at c:\b\w\e37dd45d8cd1eaf4\modules\iisintegration\src\aspnetcoremodulev2\aspnetcore\applicationinfo.cpp:91 
[aspnetcorev2.dll] Event Log: 'Failed to start application '/LM/W3SVC/717465607/ROOT', ErrorCode '0x8000ffff'.' 
End Event Log Message.

Here is my ancm.log file from Visual Studio Publish. They appear to be similar, but the module running is "aspnetcorev2_inprocess".:

[aspnetcorev2_inprocess.dll] Initializing logs for 'D:\home\site\wwwroot\aspnetcorev2_inprocess.dll'. Process Id: 13108.. File Version: 12.2.18316.0. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: ce8cf65589734f82b0536c543aba5bd60d0a5a98.
[aspnetcorev2_inprocess.dll] Waiting for initialization
[aspnetcorev2_inprocess.dll] Starting in-process worker thread
[aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: '.\[MyApp].exe' arguments: '' path: 'D:\home\site\wwwroot\'
[aspnetcorev2_inprocess.dll] Known dotnet.exe location: ''
[aspnetcorev2_inprocess.dll] Process path '.\[MyApp].exe' is not dotnet, treating application as standalone or portable with bootstrapper
[aspnetcorev2_inprocess.dll] Checking application.dll at 'D:\home\site\wwwroot\.\[MyApp].dll'
[aspnetcorev2_inprocess.dll] Checking hostfxr.dll at 'D:\home\site\wwwroot\.\hostfxr.dll'
[aspnetcorev2_inprocess.dll] hostfxr.dll found app local at 'D:\home\site\wwwroot\.\hostfxr.dll', treating application as standalone
[aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: '' hostfxr path: 'D:\home\site\wwwroot\.\hostfxr.dll' arguments:
[aspnetcorev2_inprocess.dll] Argument[0] = 'D:\home\site\wwwroot\.\[MyApp].exe'
[aspnetcorev2_inprocess.dll] In-process callbacks set
[aspnetcorev2_inprocess.dll] Event Log: 'Application 'D:\home\site\wwwroot\' started the coreclr in-process successfully.' 
End Event Log Message.
[aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'D:\home\site\wwwroot\'
[aspnetcorev2_inprocess.dll] Starting file watcher thread

Any ideas what I can do to get the correct "aspnetcorev2_inprocess.dll" ANCM module to load? Between these two installs, I do not make any changes in my azure application. Method one is:
dotnet publish -c Release --self-contained -r win10-x64
Method two is click "Publish" in Visual Studio with this:

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <WebPublishMethod>MSDeploy</WebPublishMethod>
    <PublishProvider>AzureWebSite</PublishProvider>
    <LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
    <LastUsedPlatform>Any CPU</LastUsedPlatform>
    <SiteUrlToLaunchAfterPublish>[SiteUrl]
    <LaunchSiteAfterPublish>True</LaunchSiteAfterPublish>
    <ExcludeApp_Data>False</ExcludeApp_Data>
    <UsePowerShell>True</UsePowerShell>
    <MSDeployServiceURL>[DeployServiceUrl]</MSDeployServiceURL>
    <DeployIisAppPath>[MyApp]</DeployIisAppPath>
    <RemoteSitePhysicalPath />
    <SkipExtraFilesOnServer>False</SkipExtraFilesOnServer>
    <MSDeployPublishMethod>WMSVC</MSDeployPublishMethod>
    <EnableMSDeployBackup>True</EnableMSDeployBackup>
    <UserName>[Username]</UserName>
    <_SavePWD>True</_SavePWD>
    <_DestinationType>AzureWebSite</_DestinationType>
    <ADUsesOwinOrOpenIdConnect>False</ADUsesOwinOrOpenIdConnect>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <ProjectGuid>34fc2225-66cb-43ea-bce0-8793e51fed2e</ProjectGuid>
    <SelfContained>True</SelfContained>
    <RuntimeIdentifier>win10-x64</RuntimeIdentifier>
    <InstallAspNetCoreSiteExtension>False</InstallAspNetCoreSiteExtension>
    <_IsPortable>False</_IsPortable>
  </PropertyGroup>
</Project>

I can freely deploy one then the other and back. The Visual Studio publish one consistently works, the dotnet publish one consistently does not work.

[cdwaddell]

FYI, I looked into the error reported Invalid runtimeconfig.json. That file is identical in both deployments.

[jkotalik]

Interesting. Does either the VS publish or dotnet publish output have a xxxx.runtimeconfig.dev.json file? If there is a difference there, that may be the issue.

If there isn't any difference there, can you send both the .runtimeconfig.json (and runtimeconfig.dev.json) files to me in my email listed on my github profile? There could be a Byte Order Mark (https://en.wikipedia.org/wiki/Byte_order_mark) issue.

[cdwaddell]

Ok. I found the error and the reason that it was working on Visual Studio and not in dotnet CLI. Apparently the following has been in my web.config since 2016 when I clicked file new project:

A config transform in the VS publish step turns this into:

If I change my web config manually to this value, it works after dotnet publish. and the correct "[aspnetcorev2_inprocess.dll]" module is loaded. here is the ancm.log after making this web.config change:

[aspnetcorev2.dll] Initializing logs for 'D:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 12448.. File Version: 12.2.18296.0. Description: IIS ASP.NET Core Module V2. Commit: 61f1a70784dc0a32cf98f8ddd169c0293b0390ab.
[aspnetcorev2.dll] Resolving hostfxr parameters for application: '.\ScholarRx.Identity.exe' arguments: '' path: 'D:\home\site\wwwroot\'
[aspnetcorev2.dll] Known dotnet.exe location: ''
[aspnetcorev2.dll] Process path '.\ScholarRx.Identity.exe' is not dotnet, treating application as standalone or portable with bootstrapper
[aspnetcorev2.dll] Checking application.dll at 'D:\home\site\wwwroot\.\ScholarRx.Identity.dll'
[aspnetcorev2.dll] Checking hostfxr.dll at 'D:\home\site\wwwroot\.\hostfxr.dll'
[aspnetcorev2.dll] hostfxr.dll found app local at 'D:\home\site\wwwroot\.\hostfxr.dll', treating application as standalone
[aspnetcorev2.dll] Parsed hostfxr options: dotnet location: '' hostfxr path: 'D:\home\site\wwwroot\.\hostfxr.dll' arguments:
[aspnetcorev2.dll] Argument[0] = 'D:\home\site\wwwroot\.\ScholarRx.Identity.exe'
[aspnetcorev2.dll] c:\b\w\e37dd45d8cd1eaf4\modules\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020
[aspnetcorev2.dll] Loading request handler:  'D:\home\site\wwwroot\aspnetcorev2_inprocess.dll'
[aspnetcorev2.dll] Creating handler application
[aspnetcorev2_inprocess.dll] Initializing logs for 'D:\home\site\wwwroot\aspnetcorev2_inprocess.dll'. Process Id: 12448.. File Version: 12.2.18316.0. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: ce8cf65589734f82b0536c543aba5bd60d0a5a98.
[aspnetcorev2_inprocess.dll] Waiting for initialization
[aspnetcorev2_inprocess.dll] Starting in-process worker thread
[aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: '.\ScholarRx.Identity.exe' arguments: '' path: 'D:\home\site\wwwroot\'
[aspnetcorev2_inprocess.dll] Known dotnet.exe location: ''
[aspnetcorev2_inprocess.dll] Process path '.\ScholarRx.Identity.exe' is not dotnet, treating application as standalone or portable with bootstrapper
[aspnetcorev2_inprocess.dll] Checking application.dll at 'D:\home\site\wwwroot\.\ScholarRx.Identity.dll'
[aspnetcorev2_inprocess.dll] Checking hostfxr.dll at 'D:\home\site\wwwroot\.\hostfxr.dll'
[aspnetcorev2_inprocess.dll] hostfxr.dll found app local at 'D:\home\site\wwwroot\.\hostfxr.dll', treating application as standalone
[aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: '' hostfxr path: 'D:\home\site\wwwroot\.\hostfxr.dll' arguments:
[aspnetcorev2_inprocess.dll] Argument[0] = 'D:\home\site\wwwroot\.\ScholarRx.Identity.exe'
[aspnetcorev2_inprocess.dll] In-process callbacks set
[aspnetcorev2_inprocess.dll] Event Log: 'Application 'D:\home\site\wwwroot\' started the coreclr in-process successfully.' 
End Event Log Message.
[aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'D:\home\site\wwwroot\'
[aspnetcorev2_inprocess.dll] Starting file watcher thread

I went through our CI/CD platform and confirmed that this change does fix our issue.

[cdwaddell]

Thanks @jkotalik for looking into this with me.

[pakrym]

@jkotalik can you investigate why invalid stdoutLogFile prevents apps from starting?

[jkotalik]

@cdwaddell can you confirm that changing the stdoutLogFile affects whether the app starts or not? An invalid path to a log folder shouldn't affect the app failing to start.

[copycatam]

@pakrym
Yes. invalid stdoutLogFile prevents apps from starting. I face the same problem today and changing stdoutLogFile fix it.

[pakrym]

@copycatam what was invalid about your stdoutLogFile value? Non existent path?

Would you be willing to provide a debug log when it happens following instructions from https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-2.2#enhanced-diagnostic-logs

[yangzuo0621]

Thanks. I met this problem too. And changes to "

Ok. I found the error and the reason that it was working on Visual Studio and not in dotnet CLI. Apparently the following has been in my web.config since 2016 when I clicked file new project:

A config transform in the VS publish step turns this into:

If I change my web config manually to this value, it works after dotnet publish. and the correct "[aspnetcorev2_inprocess.dll]" module is loaded. here is the ancm.log after making this web.config change:

[aspnetcorev2.dll] Initializing logs for 'D:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 12448.. File Version: 12.2.18296.0. Description: IIS ASP.NET Core Module V2. Commit: 61f1a70784dc0a32cf98f8ddd169c0293b0390ab.
[aspnetcorev2.dll] Resolving hostfxr parameters for application: '.\ScholarRx.Identity.exe' arguments: '' path: 'D:\home\site\wwwroot\'
[aspnetcorev2.dll] Known dotnet.exe location: ''
[aspnetcorev2.dll] Process path '.\ScholarRx.Identity.exe' is not dotnet, treating application as standalone or portable with bootstrapper
[aspnetcorev2.dll] Checking application.dll at 'D:\home\site\wwwroot\.\ScholarRx.Identity.dll'
[aspnetcorev2.dll] Checking hostfxr.dll at 'D:\home\site\wwwroot\.\hostfxr.dll'
[aspnetcorev2.dll] hostfxr.dll found app local at 'D:\home\site\wwwroot\.\hostfxr.dll', treating application as standalone
[aspnetcorev2.dll] Parsed hostfxr options: dotnet location: '' hostfxr path: 'D:\home\site\wwwroot\.\hostfxr.dll' arguments:
[aspnetcorev2.dll] Argument[0] = 'D:\home\site\wwwroot\.\ScholarRx.Identity.exe'
[aspnetcorev2.dll] c:\b\w\e37dd45d8cd1eaf4\modules\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020
[aspnetcorev2.dll] Loading request handler:  'D:\home\site\wwwroot\aspnetcorev2_inprocess.dll'
[aspnetcorev2.dll] Creating handler application
[aspnetcorev2_inprocess.dll] Initializing logs for 'D:\home\site\wwwroot\aspnetcorev2_inprocess.dll'. Process Id: 12448.. File Version: 12.2.18316.0. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: ce8cf65589734f82b0536c543aba5bd60d0a5a98.
[aspnetcorev2_inprocess.dll] Waiting for initialization
[aspnetcorev2_inprocess.dll] Starting in-process worker thread
[aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: '.\ScholarRx.Identity.exe' arguments: '' path: 'D:\home\site\wwwroot\'
[aspnetcorev2_inprocess.dll] Known dotnet.exe location: ''
[aspnetcorev2_inprocess.dll] Process path '.\ScholarRx.Identity.exe' is not dotnet, treating application as standalone or portable with bootstrapper
[aspnetcorev2_inprocess.dll] Checking application.dll at 'D:\home\site\wwwroot\.\ScholarRx.Identity.dll'
[aspnetcorev2_inprocess.dll] Checking hostfxr.dll at 'D:\home\site\wwwroot\.\hostfxr.dll'
[aspnetcorev2_inprocess.dll] hostfxr.dll found app local at 'D:\home\site\wwwroot\.\hostfxr.dll', treating application as standalone
[aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: '' hostfxr path: 'D:\home\site\wwwroot\.\hostfxr.dll' arguments:
[aspnetcorev2_inprocess.dll] Argument[0] = 'D:\home\site\wwwroot\.\ScholarRx.Identity.exe'
[aspnetcorev2_inprocess.dll] In-process callbacks set
[aspnetcorev2_inprocess.dll] Event Log: 'Application 'D:\home\site\wwwroot\' started the coreclr in-process successfully.' 
End Event Log Message.
[aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'D:\home\site\wwwroot\'
[aspnetcorev2_inprocess.dll] Starting file watcher thread

I went through our CI/CD platform and confirmed that this change does fix our issue.

It works for me.