Runtime: System.BadImageFormatException: 'Bad IL format.'

[Petermarcu]

@atpyk commented on Wed May 30 2018

I've upgraded dotnetcore2.0 to 2.1, and upgraded all packages too.
When I startup my web application, the exception prompts.

StackTrace:
at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at GrapeLEAF.AdminService.Program.BuildWebHost(String[] args) in E:\workspace\dotnet\XXX\XXX.AdminService\XXX.AdminService\Program.cs:line 16
at GrapeLEAF.AdminService.Program.Main(String[] args) in E:\workspace\dotnet\XXX\XXX.AdminService\XXX.AdminService\Program.cs:line 12

---

@atpyk commented on Thu May 31 2018

I uninstall original .net core sdk and runtime; and reinstall VS2017 and 2.1 sdk.
I create a new webapi application by scaffold, it doesn't work.

---

@VladislavAntonyuk commented on Thu May 31 2018

https://blogs.msdn.microsoft.com/webdev/2018/05/07/asp-net-core-2-1-0-rc1-now-available/

---

@atpyk commented on Thu May 31 2018

So sad, it doesn't work too.

---

@VladislavAntonyuk commented on Thu May 31 2018

Make sure you have installed the latest VS and try to reinstall .net core.
Roolback your changes to .net core 2.0 and follow the instruction I have sent you

---

@atpyk commented on Thu May 31 2018

I debug into Microsoft.AspNetCore.Hosting and find out the error line:
var hostingServices = BuildCommonServices(out var hostingStartupErrors);

It will throw the exception: 'Bad IL format.'

But if I rename this method to BuildCommonServices111 or other, It works.
So weird~
It seems like the namespace "Microsoft.AspNetCore.Hosting" conflicts with others, the .net runtime involves dlls of different version at same time.

---

@VladislavAntonyuk commented on Fri Jun 01 2018

make sure you have installed version 2.1.300 and try to uninstall all previous versions. Also doublecheck you have finished all steps in the instruction I've sent you

---

@atpyk commented on Fri Jun 01 2018

I have reinstall my OS, it works.
Thanks.

---

@mangod3 commented on Sat Jun 02 2018

Did you have NewRelic monitoring your .net app by any chance when you ran into the issue?

---

@Merurino commented on Sun Jun 03 2018

@mangod3 I was about to comment here. My issue seemed to be Newrelic. Was returning the IL issue. I was building it and running it on docker.

---

@mangod3 commented on Sun Jun 03 2018

@Merurino Thanks for responding. After debugging the issue it looks like the 2.1 changes related to code versioning (https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/code-versioning-profiler-breaking-changes.md) are causing issues with profiling APIs. @noahfalk who should we follow up with to get this investigated, looks like the ILHeader is not getting initialized correctly after updating a method IL using SetILFunctionBody

---

@noahfalk commented on Mon Jun 04 2018

Sorry you are running into trouble! Feel free to start with me as the investigator and I can pull in others as needed. To help get us started:
1) Could you post steps I can follow to try reproducing the problem?
2) Can you let me know what you observed while debugging that lead you to your conclusions?

---

@mangod3 commented on Mon Jun 04 2018

Thanks for your response @noahfalk. Will package up a repro and send over email.

---

@joseclament commented on Wed Jun 06 2018

I have a web job built on netcoreapp2.1 console application, targeting the win10-x64 and getting the "Unhandled Exception: System.BadImageFormatException: Could not load file or assembly exception".
I haven't had any issue with locally. Deployed using the visual studio directly to the app service. Any thoughts?

---

@noahfalk commented on Wed Jun 06 2018

@joseclament - Are you aware of any .NET profiler or monitoring agent being used in your deployment environment? If not we should probably a open a new issue because there would be a decent chance that the root cause for you will be unrelated to the root cause here.

---

@joseclament commented on Thu Jun 07 2018

@noahfalk , I am using AppInsight to push the logs, other than no profiling or tooling.
Let me know if I need to open a new issue.

---

@nil4 commented on Thu Jun 07 2018

@noahfalk I am seeing a similar issue using the AppDynamics .NET Core profiler agent on a netcoreapp2.1 application and .NET Core SDK 2.1.300.

When the application starts, the following exception is thrown:

System.BadImageFormatException
  HResult=0x8007000B
  Message=Bad IL format.
  Source=System.Private.CoreLib
  StackTrace:
   at System.AppDomainManager..ctor()
   at System.Reflection.Assembly.GetEntryAssembly()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at MyApp.Web.Startup.Main(String[] args) 

---

@joseclament commented on Thu Jun 07 2018

@noahfalk and @nil4 , I got the bottom of the problem. It took couple of days to get the bottom of the issue and made me bit nut. I was able to dig and find this link: https://tutel.me/c/programming/questions/42726350/how+to+change+azure+app+service+to+64bit

Your core should be built on x86 version even if the deploying server(Azure) is based on 64 bit architecture. Because the 64 bit .NET core processes using the .NET core runtime (as opposed to the .NET Framework runtime) are not yet supported on Azure but is planned to be coming in the future. You can verify this by go to the kudu and type dotnet --info. You can see something like this below:

.NET Core SDK (reflecting any global.json):
Version: 2.1.300
Commit: 32f29b6eb9

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x86
_Base Path: D:\Program Files (x86)\dotnet\sdk\2.1.300_ [You can see only the x86 version here]

Host (useful for support):
Version: 2.1.0
Commit: caa7b7e2ba

.NET Core SDKs installed:
1.1.8 [D:\Program Files (x86)\dotnet\sdk]
2.1.101 [D:\Program Files (x86)\dotnet\sdk]
2.1.300 [D:\Program Files (x86)\dotnet\sdk]

You may be able to debug this locally "Any CPU" option as you might have x86 and x64 based libraries installed on your machine. I could not find any documentation on MSDN about this. I was assumed it would have been automatically support x64 version. Also make sure that the application setting is enabled for the "32" bit.

---

@nil4 commented on Thu Jun 07 2018

I am not using Azure. The error occurs on my local PC when launching the application, with IIS Express, under the VS debugger, regardless of whether IIS Express 64-bit or 32-bit is selected.

The application starts up correctly if I remove the AppDynamics.Agent.Windows NuGet package, or set CORECLR_ENABLE_PROFILING environment variable to 0 in my Web.config (disabling the profiler).

---

@mangod3 commented on Thu Jun 07 2018

is it possible for you to capture a full native stack of the exception? That might help with identifying if you are hitting the same issue or something different.

---

@nil4 commented on Thu Jun 07 2018

@noahfalk Here are the steps to reproduce the issue I see. In a command prompt, run:

mkdir Repro1622 && pushd Repro1622
dotnet new web
dotnet add package AppDynamics.Agent.Windows --version 4.4.3

Open the project in Visual Studio, and build it once.

A number of files brought by the AppDynamics package show up under the project, including Repro1622.AppDynamicsConfig.json. Edit this file and replace its content with:

{
  "controller": {
    "host": "localhost",
    "port": 443,
    "account": "customer1",
    "password": "c025be84-d72d-4f6e-b5b4-b33ce264992b",
    "ssl": true,
    "enable_tls12": true
  },
  "application": {
    "name": "irrelevant",
    "tier": "irrelevant",
    "node": "irrelevant"
  }
}

These values are not important, they just need to be different than the default placeholders for the profiler to activate.

Add a Web.config file to the project, with the following content:

<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
    </handlers>
    <aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%">
      <environmentVariables>
        <environmentVariable name="CORECLR_ENABLE_PROFILING" value="1" />
        <environmentVariable name="CORECLR_PROFILER" value="{39AEABC1-56A5-405F-B8E7-C3668490DB4A}" />

        <environmentVariable name="CORECLR_PROFILER_PATH_32" value="D:\Repro1622\bin\Debug\netcoreapp2.1\AppDynamics.Profiler_x86.dll"/>
        <environmentVariable name="CORECLR_PROFILER_PATH_64" value="D:\Repro1622\bin\Debug\netcoreapp2.1\AppDynamics.Profiler_x64.dll"/>
      </environmentVariables>
    </aspNetCore>
  </system.webServer>
</configuration>

⚠️ Per the AppDynamics docs, the two CORECLR_PROFILER_PATH_* variables must be full paths; adjust as required.

Start debugging (F5) with IIS Express and note BadImageFormatException is thrown at System.AppDomainManager..ctor() when the application starts.

Find the newest .log file under %ProgramData%\AppDynamics\DotNetAgentLogs\Profiler, which should be similar to:

2018-06-07 18:22:31.086553<info>:Use CoreCLR profiler
2018-06-07 18:22:31.089553<info>:Should instrument standalone profiler process (config found at D:\Repro1622\bin\Debug\netcoreapp2.1\Repro1622.AppDynamicsConfig.json): dotnet as app Repro1622.dll
2018-06-07 18:22:31.092555<info>:Agent Configuration: {"controller":{"account":"customer1","enableTls12":true,"ssl":true,"host":"localhost","password":"***","port":443},"application":{"tier":"irrelevant","node":"irrelevant","name":"irrelevant"}}
2018-06-07 18:22:31.092555<info>:Enabled rejit normally, disabled all NGEN images.
2018-06-07 18:22:31.163553<error>:InjectMEE failed.
2018-06-07 18:22:31.935346<info>:Transforming 57540:clrhost:System.Void System.AppDomainManager..ctor() with [[],[]]

---

@noahfalk commented on Thu Jun 07 2018

@joseclament - Glad you were able to get the bottom of it and very appreciated that you came back to post your findings in case others hit the same issue in the future! Indeed it looks like your issue had a similar symptom but different root cause than the other ones here. I created dotnet/coreclr#1670 to track any further work there.

@nil4 - Thanks for the repro! Let me try repro your issue so we can figure out if this is the same root cause or something else.

---

@noahfalk commented on Thu Jun 07 2018

@nil4 - Thanks, I was able to reproduce the problem easily using your excellent instructions! Although the AppDynamics issue is slighly different than the NewRelic one (AppDynamics does not call ICorProfilerInfo::SetILFunctionBody or ICorProfilerFunctionControl::SetILFunctionBody) , the fix I submitted earlier today (#18322) does solve both issues at the same time.

I wasn't sure if you were an employee with AppDynamics or a customer of AppDynamics? For potential short term resolutions to the issue you could either continue to use .Net Core 2.0 or use a daily build of .Net Core that has this fix as soon as the installers are ready for download (I'll write back again when they are). A little further in the future I'm attempting to get this fix into the official servicing releases for 2.1. I can also try to reach out to AppDynamics to see if there are any additional solutions available for the issue.

Let me know if you've got any other questions or concerns?

---

@nil4 commented on Fri Jun 08 2018

@noahfalk glad I could help, and thank you for the update, much appreciated! I am a customer, and have opened an AppDynamics support ticket about this; will update it shortly with a link to this issue.

[Petermarcu]

@noahfalk is this something we should fix in 2.1 or is this a vNext fix?

[noahfalk]

I think we want it in 2.1. I was just attempting to get our ducks in a row testing wise before bringing it up in shiproom.

[mangod3]

Yes please hotfix 2.1 with this change, a lot of customers are hitting this already.

[noahfalk]

Following up for anyone looking for daily 2.2 preview builds with the fix. The fix is present in Microsoft.NETCore.App version >= 2.2.0-preview1-26610-01. Download runtime daily builds here.

The fix is not yet present in any version of the complete SDK installers. As of the latest SDK, 2.2.100-preview1-009015, it is still bundling the ever so slightly older 2.2.0-preview1-26609-02 runtime.

If you don't want to wait for a complete SDK that bundles the fix you could force it forward yourself:
1) Install the 2.2.100-preview1-009015 SDK
2) Remove the 2.2.0-preview1-26609-02 runtime that came with it (delete the folder C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0-preview1-26609-02)
3) Install the latest runtime build instead

Separately from this I'm working on a hotfix for 2.1.

[sayfulloev]

Looking forward for this fix, as we got hit by it.

[nycdotnet]

Noah we're looking forward to the 2.1 hot fix at Namely. We're on .NET Core 2.0 in prod for many of our services, and we use New Relic. With yesterday's blog post that .NET Core 2.0 will be out of support in ~70 days, we're quite nervous that we will find ourselves in a difficult position soon. Thanks for your work on this issue.

[richlander]

.NET Core 2.0 EOL date has been extended to October 1, 2018 due to this issue.

See: https://twitter.com/dotnet/status/1010269960948465665

[noahfalk]

Further follow up for anyone looking for daily 2.2 preview builds. The fix is present in both daily runtime and SDK builds at this point: https://github.com/dotnet/core/blob/master/daily-builds.md.

2.1 servicing is still in progress. Part way through we discovered a 2nd issue (#18602) when using the Visual Studio debugger disassembly window. Although we determined it was pre-existing bug in 2.0 and 2.1, the user visible behavior became worse when combined with the fix for this issue. That 2nd issue now has a fix checked in for daily builds and the 2.1 servicing will include both fixes.

[rhyous]

I wonder if bug #27907 is related.

[nycdotnet]

We struggled a bit finding the right one, but we're going to try deploying with the .tar.gz for Linux x64 in the "master" column (https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-sdk-latest-linux-x64.tar.gz) from here to see if we can get this going now. It has the 3.x banner, which is weird, but we assume this should have the fix. Thanks so much to everyone involved.

[noahfalk]

I wonder if bug dotnet/coreclr#27907 is related.

@rhyous - The issue here is in code that is unique to .Net Core and if I understand 27907 correctly it occurs on desktop .NET Framework. I don't believe they are related.

It has the 3.x banner, which is weird

At some point recently I think the 2.2 work went to its own branch and the daily builds are now labeled 3.x. If the issue still persists in that build please let me know and I'll help you get to the bottom of it.

[nycdotnet]

We were indeed able to get Newrelic running on 95c6eff6daa1a69f29c42b2d405400ad44bdec91 3.0.100-preview1-009020. 😄
cc: @abe545

[noahfalk]

Thanks for the confirmation!

[abhinavgalodha]

We are also having the same issue with AppDynamics Instrumentation. Our application is on asp.net core 2.1 in beta and moving to production next week. Any ideas about the 2.1 Servicing timeline for this issue will help us plan in advance.

[noahfalk]

The goal for 2.1.3 servicing release is to ship early to mid August with daily builds potentially available earlier. I reached out to AppDynamics in case they have any additional guidance they want to pass along.

[abhinavgalodha]

Thanks @noahfalk for the update.

[Eilon]

@noahfalk - a customer who was hitting a similar issue to this in https://github.com/aspnet/KestrelHttpServer/issues/2680, and it seems like resolving this issue here will address that. I think the clock is ticking for 2.1.3 - do you know if this is still planned to make it in?

[noahfalk]

@Eilon - The changes went into the 2.1.3 release branch a few weeks ago: https://github.com/dotnet/coreclr/commit/1cebf4d46325b40591adb8a0f914f0029e48fb57

I was hoping to point to an official daily servicing build of 2.1 for developers to try out, but so far I don't think we have those builds being published.

[Eilon]

@noahfalk got it, thanks for the update!

[mviswana]

Huge thanks to @noahfalk for proactively reaching out to AppDynamics engineering and partnering with us on these fixes. We had a productive conversation which I believe will help us get robust support for .NET Core 2.1 out the door. Specifically, we are working on uptaking the async/await and ASP.NET Core 2.1 changes. Look for feature updates coming soon. Please reach out again if you have further questions on GitHub or the AppDynamics Community.

Meera Viswanathan
Senior Product Manager, AppDynamics - .NET And Azure

[peacecwz]

I couldn't track this issue. This PR is related this issue? https://github.com/dotnet/coreclr/pull/18322 I'm wondering this issue @noahfalk

[mangod3]

Is there a more precise ETA for a 2.1 fix to be released?

[martincostello]

You can use the Early Access Downloads if you can't wait for the final release: https://github.com/aspnet/Home/wiki/.NET-Core-2.1.3-Early-Access-Downloads

[peacecwz]

@martincostello Actually we are waiting New Relic for .NET Core 2.1 and they are waiting new version of .NET Core. It will be release soon?

[martincostello]

@peacecwz I'm under the impression it will ship before the end of next week.

[ghost]

Looks like the August release of the SDK is out (v2.1.4) but it looks like includes coreclr v 2.1.2 still. Does this mean we will have to wait for the September release? Or is there another release coming?

[noahfalk]

@rv-calbert - I just looked at our release page and I agree, that is not what I had expected to see. I'm not tight in the loop with the devs that handle the release mechanics but if you hang tight I'm trying to get to the bottom of it. I'll let you know as soon as I hear back from them.

[martincostello]

I think 2.1.3 is coming with SDK 2.1.401, which hasn’t been formally released yet.

[peacecwz]

Why .NET Core 2.1.3 won't be released? If you can't release it before Eid al-Adha 😄

[noahfalk]

Release team got back to me and said they are aiming for 2.1.3 next Tuesday. So its still this month, not September. Sorry for the confusion.

[leecow]

Yep, we're priming the pump to get 2.1.3 with the 2.1.401 SDK out to you all on Tuesday.

[peacecwz]

.NET Core 2.1.3 is released. This issue was solved? Why is this issue open?

[ZackTech]

I am dynamically loading .net standard dll's using MEF on Azure and I am getting Bad IL Format, on my local PC everything works fine. The MEF dlls used to load OK. Also seems to happen on our local IIS server now. Any ideas how I fix the problem? This is an urgent issue for me!

[mikey0000]

Probably need to wait for general availability of 2.1.3 in azure

On 24 August 2018 03:02:08 GMT+12:00, ZackTech [e-mail] wrote:

I am dynamically loading .net standard dll's using MEF on Azure and I
am getting Bad IL Format, on my local PC everything works fine. The MEF
dlls used to load OK. Also seems to happen on our local IIS server
now. Any ideas how I fix the problem? This is an urgent issue for me!

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/dotnet/coreclr/issues/18448#issuecomment-415450200

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[noahfalk]

With the release last week of CoreCLR 2.1.3 I'm closing this issue. As best I can tell from my own past testing and reports from the community it does resolve the exception. If there are cases that still reproduce this exception after installing 2.1.3 I'd recommend opening a new issue. Sorry for the trouble this one caused.