Autorest: [.NET] AmbiguousMatchException on initializing ServiceClient class

[OlehUdovytskyi]

Hi guys.

I have an issue with initializing my service client created by AutoRest and inherited from ServiceClient.

It throws AmbiguousMatchException and it arises in ServiceClient class.

[AmbiguousMatchException: Multiple custom attributes of the same type found.]
   System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit) +122
   Microsoft.Rest.ServiceClient`1.get_FrameworkVersion() +103
   Microsoft.Rest.ServiceClient`1.SetDefaultUserAgentInfo() +177
   Microsoft.Rest.ServiceClient`1.SetUserAgent(String productName, String version) +50
   Microsoft.Rest.ServiceClient`1..ctor(DelegatingHandler[] handlers) +61

It only appears on my Remote machine with Windows Server 2008 R2 Standart .NET Framework 4.6.1. Locally on Windows 10 Anniversary 4.6.2 Framework everything is ok.

In Microsoft.Rest.ServiceClient I've found next part of code in ClientVersion property getter:

          {
            AssemblyInformationalVersionAttribute customAttribute1 = assembly.GetCustomAttribute(typeof (AssemblyInformationalVersionAttribute)) as AssemblyInformationalVersionAttribute;
            this._clientVersion = customAttribute1 != null ? customAttribute1.InformationalVersion : (string) null;
            if (string.IsNullOrEmpty(this._clientVersion))
            {
              AssemblyFileVersionAttribute customAttribute2 = assembly.GetCustomAttribute(typeof (AssemblyFileVersionAttribute)) as AssemblyFileVersionAttribute;
              this._clientVersion = customAttribute2 != null ? customAttribute2.Version : (string) null;
            }
          }
          catch (AmbiguousMatchException ex)
          {
          }

I think it is source of my problem but I don't understand why it occurs and to handle it. I see that current exception must be caught but I don't have any other ideas what else can be wrong.

Guys, any ideas ?

[shaunol]

I'm also having this same problem. Working fine on my Windows 10 development workstation but when I deploy to my Server 2012 environment, I get the above exception. I also can't seem to find the source code for the Microsoft.Rest.ClientRuntime package to try to debug. I can't really make heads or tails of this issue.

[shaunol]

Downgrading Microsoft.Rest.ClientRuntime from 2.3.3 to 2.3.2 has fixed the issue for me.

[opterios]

I had exactly the same issue with shaunol (on Windows 8 it was fine, on Windows 2012 it was not working) while trying to make calls to AzureBatch. I was getting the following exception:

Message='Multiple custom attributes of the same type found. at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit)
 at Microsoft.Rest.ServiceClient1.get_FrameworkVersion()
 at Microsoft.Rest.ServiceClient1.SetDefaultUserAgentInfo()
 at Microsoft.Rest.ServiceClient1.SetUserAgent(String productName, String version)
 at Microsoft.Rest.ServiceClient1.InitializeHttpClient(HttpClientHandler httpClientHandler, DelegatingHandler[] handlers)


and downgrading to 2.3.2 resolved the issue!

Thanks for your help!

[robpex]

Adding info from my troubleshooting this problem.

In my case, the "Application Insights Monitor" (https://github.com/Azure/azure-content/blob/master/articles/application-insights/app-insights-monitor-performance-live-website-now.md) was injecting an additional "AssemblyFileVersionAttribute" value into mscorlib.dll. So, the Microsoft.Rest.ClientRuntime error makes sense because when it tries to get the file version using that attribute, it is, in fact, ambiguous.

1) Does it ever make sense to have more than one "AssemblyFileVersionAttribute" on any file? If not, there's at least a bug in the tool I just mentioned.

2) Regardless of the answer to number 1, you'll see that the code for Microsoft.Rest.ClientRuntime tries to use "AssemblyFileVersionAttribute" more than once. In one instance, it catches and ignores AmbiguousReferenceException and in other instances, it is not. I believe it should be consistent. Either it's always safe to ignore it, or it's not.

[shaunol]

Noting that I also have Application Insights Monitor installed on my Server 2012 instance that is exhibiting this behaviour. Really good catch.

[SergeyKanzhelev]

@robplex, Application Insights do not inject any attributes into mscorlib. What made you make this conclusion?

Application Insights do make some code injections. But I cannot think of any that will make the custom attributes reading logic to fail. Do you have a small repro I can try out?

[robpex]

@SergeyKanzhelev the way I reproduced it was by debugging my code and at the breakpoint (I don't think the breakpoint location is important), running the following in an immediate window:
System.Attribute.GetCustomAttributes((typeof(object)).Assembly).

From there, I was able to see multiple {System.Reflection.AssemblyFileVersionAttribute} with different values and when I uninstall Application Insights, this was no longer the case. I'm sure there are other ways of getting at those properties for mscorlib, but this is what caused me to make this conclusion.

I saw Microsoft in your profile...are you directly involved with Application Insights?

[SergeyKanzhelev]

@robpex yes, I am working on Application Insights. Do you happen to have a small repro I can try out? I'll check the collection System.Attribute.GetCustomAttributes((typeof(object)).Assembly) on my apps now and report back.

[SergeyKanzhelev]

Ok. I got a repro. We will investigate

[SergeyKanzhelev]

BTW, this is the magic we are trying to make with the status monitor if you are interested: http://apmtips.com/blog/2016/11/18/how-application-insights-status-monitor-not-monitors-dependencies/

If you want to keep using application insights - you can disable profiler and only use Application Insights SDK. It will still collect interesting telemetry. Let me know if you need help configuring it

[robpex]

@SergeyKanzhelev ok, cool. I can't share the code (company policies), but I'm happy to discuss further since I dug pretty deep on this one. :) I suspect there's still something to be fixed in the autorest side as well as I mentioned above in number 2. My initial thought was to catch/ignore the exception like it's doing elsewhere, but I don't know if that's the best approach. I don't know if you're involved with this project though, but perhaps you know someone. I wasn't sure who to tell. :)

[SergeyKanzhelev]

I looked at the list of contributors and I only know one person there. I think the issue is clearly related to Application Insights and needs to be fixed on our side. After that no additional exception handling will be needed. But I didn't dug deep, just initial thoughts

[brjohnstmsft]

Possibly related: https://github.com/Azure/azure-sdk-for-net/issues/2516

[brjohnstmsft]

@SergeyKanzhelev Are you investigating this issue? I'm getting more reports of customers being affected.

[SergeyKanzhelev]

We know the root cause and working on a fix. We expect the fix shipped next week. Meanwhile the work around is to disable Application Insights Azure WebSite extension or StatusMonitor. Please escalate using CSS channel if you need a hotfix sooner.

[jimt4593]

Status Monitor somehow causes AmbiguousMatchException exception on AutoRest apps #241 is the issue that @SergeyKanzhelev is referring to.

The issue is marked closed and mentions Application Insights 2.2.0. Can someone confirm that updating to Application Insights 2.2.0 fixes this issue?

[SergeyKanzhelev]

@jimt4593 the issue you mentioned is still open. We released a fix fro Azure Web Apps and working on on-premisis Status Monitor.

We will also appreciate a confirmation that the fix actually worked.

[vdevappa]

Never worked. Only way I could get it to work is to have a new asp.net project which does not use app insights and when deploying the cloud service - do not enable sending telemetry. If you do, error still happens.

SOMEONE FIX THIS SOON PLEASE!!!

[arnaudauroux]

Any update on this @SergeyKanzhelev ? I am also experiencing the same issue with the Azure Search Library and Asp.Net Core
"Microsoft.Azure.Search": "3.0.1",
"Microsoft.ApplicationInsights.AspNetCore": "1.0.2",

[vdevappa]

These bozos at Microsoft do not reply. They screw everyone up and they shut their mouths and we are suffering because of it - marketing says - hey! everyone c'mon use our services!

Guys, don't keep referencing these s!@#$%y links which take you on a merry go around. We don't have the time to crawl the entire internet trying to figure out how to fix the s#$t you broke.

This is supposed to be enterprise search capability - not corner mom & pop shop search which you can break like this and keep broken for 17 #@#$%^& days.

Anyway...

What I ended up doing - this is the only way to get this c@#p to work - created another project - without application insights referenced in it and then put my search code there and it works.

If you remove, uninstall app insights packages - still does not work - I already tried it.

Only caveat, the wizard keeps prompting to select the app insights when you try to deploy the cloud service everytime - but I can live with that.

[arnaudauroux]

@vdevappa I have downgraded Microsoft.Azure.Search from 3.0.1 to 1.1.3 and it solved the problem ;)

[vdevappa]

Ya, not a great solution I think to downgrade search for a reason which is not even remotely related to it (app insights). I went the other way.

[SergeyKanzhelev]

@vdevappa @arnaudauroux in what environment you are running the application? You don't need to know these details, but there are two pieces of Application Insights - SDK that you install with the application and agent that you install. In Azure Web Sites - agent is deployed by WebSite extension, in Cloud Services by WAD and in other cased for regular IIS - by Application Insights Status Monitor. We updated extension 17 days ago and Status Monitor yesterday. WAD update is pending as we need to test is as well.

So you'd need to update an agent, not only uninstall Application Insights. I still didn't get confirmation from anybody that the issue was actually fixed after update and will appreciate if you'll try it out.

[vdevappa]

I use cloud services. I know the last release did not work as you mention (WAD) because it is pending.

[SergeyKanzhelev]

In cloud services - did you enable Application Insights using WAD (Diagnostics Settings window in Visual Studio) or using a startup task?

[arnaudauroux]

I use Azure Web App

[vdevappa]

In cloud services, Application Insights was enabled a weeks ago (before I implemented search) using right click > configure/ install application insights.

[SergeyKanzhelev]

@vdevappa - can you check whether you have Application Inisghts Sink configured in wadcfg file. See this

@arnaudauroux - in the list of azure Web App extensions you should have an Application Insights. And it should say that update is available.

[vdevappa]

I see the config in the diagnostics.wadcfgx file. No instrumentation key in the below config though.

<SinksConfig> <Sink name="applicationInsights"> <ApplicationInsights /> <Channels> <Channel logLevel="Error" name="errors" /> </Channels> </Sink> </SinksConfig>

[SergeyKanzhelev]

Please remove this sink completely and try it. I believe instrumentation key will be substituted on deployment.

[vdevappa]

I can't do this right now. Already moved on and put search in another service.

[SergeyKanzhelev]

@vdevappa thanks!

@arnaudauroux have you had a chance to try it with the updated extension?

[arnaudauroux]

@SergeyKanzhelev The update failed with the following error trace:

Failed to update web app extension Application Insights. {"Message":"ExitCode: 1, Output: 1/9/2017 9:17:10 AM: [Error] Failure happened during instrumentation: Application Insights extension doesn't support ASP.NET Core applications. Please enable Application Insights through the application project code., 
Error: , Kudu.Core.Infrastructure.CommandLineException: 
\r\nD:\\home\\SiteExtensions\\Microsoft.ApplicationInsights.AzureWebSites\\install.cmd \r\n   
at Kudu.Core.Infrastructure.Executable.ExecuteInternal(ITracer tracer, Func`2 onWriteOutput, Func`2 onWriteError, Encoding encoding, String arguments, Object[] args)\r\n   
at Kudu.Core.Infrastructure.Executable.ExecuteWithProgressWriter(ILogger logger, ITracer tracer, String arguments, Object[] args)\r\n   
at Kudu.Core.SiteExtensions.SiteExtensionManager.<>c__DisplayClass25_1.<InstallExtension>b__1()\r\n   
at Kudu.Core.Infrastructure.OperationManager.<>c__DisplayClass2_0.<Attempt>b__0()\r\n   
at Kudu.Core.Infrastructure.OperationManager.Attempt[T](Func`1 action, Int32 retries, Int32 delayBeforeRetry, Func`2 shouldRetry)\r\n   
at Kudu.Core.Infrastructure.OperationManager.Attempt(Action action, Int32 retries, Int32 delayBeforeRetry)\r\n   
at Kudu.Core.SiteExtensions.SiteExtensionManager.<InstallExtension>d__25.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n  
at Kudu.Core.SiteExtensions.SiteExtensionManager.<TryInstallExtension>d__24.MoveNext()"}

The extension update mechanism does not apparently work with dotnet core.. :(

[SergeyKanzhelev]

@arnaudauroux this is a known limitation. I didn't realize you are using .NET Core. Anyway update of the extension will bring a new "agent" piece that will eliminate the AmbiguousMatchException problem. Is it the case for you?

[vdevappa]

Guys, I have an update here. The same error happens when application insights is installed in the cloud service web api project if you try to use any version of the MongoDB.Driver 2.4.0 or beyond.

Message="System.Reflection.AmbiguousMatchException: Multiple custom attributes of the same type found. at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit) at System.Runtime.InteropServices.RuntimeInformation.get_FrameworkDescription() at System.Lazy1.CreateValue()--- End of stack trace from previous location where exception was thrown --- at System.Lazy1.get_Value() at MongoDB.Driver.Core.Connections.ClientDocumentHelper.CreateClientDocument(String applicationName) at MongoDB.Driver.Core.Connections.BinaryConnectionFactory..ctor(ConnectionSettings settings, IStreamFactory streamFactory, IEventSubscriber eventSubscriber) at MongoDB.Driver.Core.Configuration.ClusterBuilder.BuildCluster() at MongoDB.Driver.ClusterRegistry.CreateCluster(ClusterKey clusterKey) at MongoDB.Driver.ClusterRegistry.GetOrCreateCluster(ClusterKey clusterKey) at MongoDB.Driver.MongoClient..ctor(MongoClientSettings settings) at Locus.MongoDB.Client.Initialize(String databaseName, String hostName, Int32 hostPortNo, String userName, String password) at Locus.MongoDB.LocusConfig.LocusConfigData.Database() at Locus.MongoDB.MongoConfigReader.GetConfiguration[T](String collectionName, Func2 getDefaultConfig) at Locus.Configuration.CommonDbConfig.GetLoggingConfiguration()" TraceSource="w3wp.exe"`

This is a total disaster. Now, I can't even use MongoDB because application insights won't play well with it.

Any hope of a fix anytime in the near future?

Removing the SinksConfig completely did not make any difference.

Regards,
Vijay

[ishankalakshan]

@SergeyKanzhelev "We expect the fix shipped next week" Is this fix already there? We are trying to use some feature (analysers) of azure search 3.0. So there is no way to downgrade libraries. Still it works only in local environment.When deployed to azure search throws the AmbiguousMatchException.

[SergeyKanzhelev]

@Brahmnes was WAD update shipped?

[tbolon]

Hi,

We had the same problem on PowerBI Embeded. This package is using Microsoft.Rest.ClientRuntime. The package was updated to version 2.3.4 which have this problem.

I was able to restore the app by downgrading the package to 2.3.2.

This package source seems not to be hosted on github.

See stacktrace below:

System.Reflection.AmbiguousMatchException: Multiple custom attributes of the same type found.
   at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit)
   at Microsoft.Rest.ServiceClient`1.get_FrameworkVersion()
   at Microsoft.Rest.ServiceClient`1.SetDefaultUserAgentInfo()
   at Microsoft.Rest.ServiceClient`1.SetUserAgent(String productName, String version)
   at Microsoft.Rest.ServiceClient`1.InitializeHttpClient(HttpClientHandler httpClientHandler, DelegatingHandler[] handlers)
   at Microsoft.PowerBI.Api.V1.PowerBIClient..ctor(ServiceClientCredentials credentials, DelegatingHandler[] handlers)

[SergeyKanzhelev]

@tbolon, what environment your application is deployed? I just got confirmation that WAD agent got updated so the bug should be gone in most environments.

• Cloud Services - WAD agent will be updated with the next deployment
• Azure Web Site - you need to update extension as shown on picture below
• Status Monitor - update via Web Platform Installer

[keremdemirer]

@SergeyKanzhelev Do we get the updated extension if we re-reploy the application from visual studio or vsts?

[SergeyKanzhelev]

@keremdemirer yes, unless you are creating a new application when deploying. Unfortunately Azure Web Apps extensions doesn't support automatic update

[keremdemirer]

@SergeyKanzhelev I'm sorry, forgot to mention. I deploy to Cloud Service web and worker roles. Does the same applies?

[SergeyKanzhelev]

@keremdemirer , do you update or redeploy and VIP swap? I believe the agent will be updated on redeploy, not on update, +@brahmnes to confirm

[brahmnes]

A new WAD update hasn't shipped yet, hopefully we can update later this week.
Updating the agent won't help until that has happened.

[tbolon]

@SergeyKanzhelev You are right, extension was out of date. We have updated them to the latest revision (we are using azure web site). Thanks.

[YZahringer]

Same exception with v2.3.5 on UWP application with Xamarin. No problem with v2.3.2.

Only in Release mode with .NET Native tool chain activated.

[SergeyKanzhelev]

The root cause of original problem was the Application Insights profiler that will inject extra custom attributes. We do not enable profiler for UWP apps so the root cause should be different. Maybe Native tool chain is doing something similar to what we do for Application Insights

Does this code returns multiple values for you as well?

System.Attribute.GetCustomAttributes((typeof(object)).Assembly)

What attributes will this code return:

Console.WriteLine("Number of custom attributes:" + System.Attribute.GetCustomAttributes((typeof(object)).Assembly).Length);

foreach(var a in System.Attribute.GetCustomAttributes((typeof(object)).Assembly))
{
    Console.WriteLine("attribute: " + a.TypeId);
}

[peters]

Same defect can be observed using the MongoDB driver 2.4.2 driver:
https://jira.mongodb.org/plugins/servlet/mobile#issue/CSHARP-1863

[peters]

Apparently uninstalling the application insights toolkit resolves the issue. The version installed on the servers was pretty old and since we are bundling application insights with our web app it was never required anyway.

[mpbelov]

@SergeyKanzhelev I faced with this issue using MongoDB driver. As I undesrtood, the issue was resolved and Azure services were updated with the fixed version of AppInsights. However I am still experiencing it. I've tried to redeploy my cloud service, delete and re-create it from scratch, update my VS components. Nothing helped. Only switching off Azure diagnostics when deploying resolved it. How to get fixed version of monitoring tools for cloud services?

[SergeyKanzhelev]

@mpbelov I just checked with Cloud Service team. They said they reverted WAD changes due to another issue and deployed it back ~2/13/2017 4:01 PM PCT. I'm really sorry you wasted your time trying to get it fixed when it was reverted back.

[SergeyKanzhelev]

@peters do you know which custom attributes MongoDB driver is complaining about? As far as I know the only attributes was fixed is AssemblyVersion as we needed to do it faster. We will have a version that doesn't mess up with attributes at all later

[peters]

@SergeyKanzhelev AssemblyFileVersionAttribute

[SergeyKanzhelev]

@peters this should be fixed. Thanks!

[maxvella]

The uwp problem does not appear to have been resolved. Will work as expected in Debug mode but then throws an AmbiguousMatchException when compiled using the .Net Native toolchain even when code optimization is turned off. This precludes apps relying on this package from being distributed via the Windows Store. I encountered the problem with Azure Search SDK 3.0.2 which has a dependency on it. Any possible fix in sight?

[SergeyKanzhelev]

@sergiy-k, @morganbr, Azure Search module uses assembly attributes to get the version of the framework. It threw Ambiguios exception in case when we injected code into mscorlib (which we fixed). Now something wrong with .NET Native.

Can you please investigate?

[maxvella]

@SergeyKanzhelev thank you for the prompt reply. This is the stack trace of the AmbiguousMatchException that I'm getting:

   at System.Reflection.CustomAttributeExtensions.OneOrNull[T](IEnumerable`1 results) in f:\dd\ndp\fxcore\src\System.Reflection.Extensions\System\Reflection\CustomAttributeExtensions.cs:line 318
   at System.Reflection.CustomAttributeExtensions.GetCustomAttribute(Assembly element, Type attributeType) in f:\dd\ndp\fxcore\src\System.Reflection.Extensions\System\Reflection\CustomAttributeExtensions.cs:line 67
   at Microsoft.Rest.ServiceClient`1.get_FrameworkVersion()
   at Microsoft.Rest.ServiceClient`1.SetDefaultUserAgentInfo()
   at Microsoft.Rest.ServiceClient`1.SetUserAgent(String productName, String version)
   at Microsoft.Rest.ServiceClient`1.InitializeHttpClient(HttpClientHandler httpClientHandler, DelegatingHandler[] handlers)
   at Microsoft.Rest.ServiceClient`1..ctor(HttpClientHandler rootHandler, DelegatingHandler[] handlers)
   at Microsoft.Rest.ServiceClient`1..ctor(DelegatingHandler[] handlers)
   at Microsoft.Azure.Search.SearchIndexClient..ctor(DelegatingHandler[] handlers)
   at Microsoft.Azure.Search.SearchIndexClient..ctor(String searchServiceName, String indexName, SearchCredentials credentials)
   at XXXX.CloudServices.SearchClient..ctor(String searchServiceName, String searchIndexName, String apiKey) in D:\Programming\XXXX\XXXX.CloudServices\SearchClient.cs:line 32

As I said in my previous post, the exception is only raised when compiling against the .Net Native tool chain. Turning code optimization off makes no difference. When I run against the .Net Framework everything works as expected.

I could only by-pass the issue by downgrading Azure Search to version 1.1.3. That seems to work on my local machine. This seems to be a short-lived solution, however, because the package is also failing WACK certification since "The binary Microsoft.Rest.ClientRuntime.dll is built in debug mode." and that "The binary Microsoft.Rest.ClientRuntime.Azure.dll is built in debug mode."

Not sure whether this is related, but the problems I am experiencing always seem to lead back to the ClientRuntime.dll

[SergeyKanzhelev]

@maxvella can you please run this code to get the list of dups. Maybe it will give an idea of what's going on... Maybe modify it to output the value of the attribute as well:

Console.WriteLine("Number of custom attributes:" + System.Attribute.GetCustomAttributes((typeof(object)).Assembly).Length);

foreach(var a in System.Attribute.GetCustomAttributes((typeof(object)).Assembly))
{
    Console.WriteLine("attribute: " + a.TypeId);
}

I was told that @sergiy-k, @morganbr are the right people for .NET Native questions. This investigation step will definitely help.

[brjohnstmsft]

Not sure whether this is related, but the problems I am experiencing always seem to lead back to the ClientRuntime.dll

@maxvella The FrameworkVersion property that throws the exception is implemented in ClientRuntime. Versions of the Azure Search SDK prior to 3.x depend on an older version of ClientRuntime that doesn't have this property, so that's why downgrading works for you.

[morganbr]

@SergeyKanzhelev , we can probably help with that. Since there's been a lot of iteration in this thread, can you help us understand the current state a bit? In particular:

1. Which attribute is the SDK looking for?
2. Which assemblies is it looking on?
3. What version of Visual Studio are you testing with?

[SergeyKanzhelev]

@morganbr to the best of my knowledge - Azure Search SDK tries to get the framework version by looking at AssemblyFileVersionAttribute of the mscorlib (actually of the ((typeof(object)).Assembly. It throws if there are multiple attributes if this type.

We caused having multiple attributes in cloud services as we injected some code into mscorlib in runtime. I do not know what may cause having multiple attributes of this type in .NET Native.

I do not have a repro with .NET Native. Maybe @maxvella can help with repro and VS version.

[brjohnstmsft]

@SergeyKanzhelev To clarify, it is the ClientRuntime library that is shared by many Azure SDKs that looks at AssemblyFileVersionAttribute. The Azure Search library is but one of many libraries depending on ClientRuntime that could potentially experience this problem.

[mms-]

Same issue on UWP and my own api client. Creating an instance using Microsoft.Rest.ServiceClient(params DelegatingHandler[] handlers) gives the following when using .Net Native Toolchain in Debug or Release:

'TTStation.exe' (Win32): Loaded 'C:\Windows\SysWOW64\version.dll'. Skipped loading symbols. Module is native, and native debugging is currently disabled.
'TTStation.exe' (Win32): Loaded 'C:\Windows\SysWOW64\Windows.Graphics.dll'. Skipped loading symbols. Module is native, and native debugging is currently disabled.
Exception thrown: 'System.Reflection.AmbiguousMatchException' in System.Reflection.Extensions.dll
'TTStation.exe' (Win32): Loaded 'C:\Windows\SysWOW64\xmllite.dll'. Skipped loading symbols. Module is native, and native debugging is currently disabled.
'TTStation.exe' (Win32): Unloaded 'C:\Windows\SysWOW64\wshbth.dll'
'TTStation.exe' (Win32): Unloaded 'C:\Windows\SysWOW64\FWPUCLNT.DLL'
Exception thrown: 'System.Reflection.AmbiguousMatchException' in System.Reflection.Extensions.dll
Exception thrown: 'System.Reflection.AmbiguousMatchException' in System.Private.CoreLib.dll
TTSessionSink.StartValidation Ex: System.Reflection.AmbiguousMatchException: Ambiguous match found.
   at SharedLibrary!<BaseAddress>+0x49a895
   at SharedLibrary!<BaseAddress>+0x8cdcde
   at Microsoft.Rest.ServiceClient<TT.Apis.v2.TTAPIv2>.get_FrameworkVersion()
   at Microsoft.Rest.ServiceClient<TT.Apis.v2.TTAPIv2>.SetDefaultUserAgentInfo()
   at Microsoft.Rest.ServiceClient<TT.Apis.v2.TTAPIv2>.SetUserAgent(String productName, String version)
   at Microsoft.Rest.ServiceClient<TT.Apis.v2.TTAPIv2>.InitializeHttpClient(Net.Http.HttpClientHandler httpClientHandler, Net.Http.DelegatingHandler[] handlers)
   at Microsoft.Rest.ServiceClient<TT.Apis.v2.TTAPIv2>..ctor(Net.Http.HttpClientHandler rootHandler, Net.Http.DelegatingHandler[] handlers)
   at Microsoft.Rest.ServiceClient<TT.Apis.v2.TTAPIv2>..ctor(Net.Http.DelegatingHandler[] handlers)

The code you provided does not compile under UWP, but

```c#
System.Diagnostics.Debug.WriteLine("Number of custom attributes:" + typeof(object).GetTypeInfo().Assembly.GetCustomAttributes().Count());

foreach (var a in typeof(object).GetTypeInfo().Assembly.GetCustomAttributes().OrderBy(o => o.GetType().GetTypeInfo().FullName))
{
System.Diagnostics.Debug.WriteLine("attribute: " + a.GetType().GetTypeInfo().FullName);
}

Number of custom attributes:28
attribute: System.CLSCompliantAttribute
attribute: System.CLSCompliantAttribute
attribute: System.Diagnostics.DebuggableAttribute
attribute: System.Diagnostics.DebuggableAttribute
attribute: System.Reflection.AssemblyCompanyAttribute
attribute: System.Reflection.AssemblyCompanyAttribute
attribute: System.Reflection.AssemblyCopyrightAttribute
attribute: System.Reflection.AssemblyCopyrightAttribute
attribute: System.Reflection.AssemblyDefaultAliasAttribute
attribute: System.Reflection.AssemblyDefaultAliasAttribute
attribute: System.Reflection.AssemblyDescriptionAttribute
attribute: System.Reflection.AssemblyDescriptionAttribute
attribute: System.Reflection.AssemblyFileVersionAttribute
attribute: System.Reflection.AssemblyFileVersionAttribute
attribute: System.Reflection.AssemblyInformationalVersionAttribute
attribute: System.Reflection.AssemblyInformationalVersionAttribute
attribute: System.Reflection.AssemblyMetadataAttribute
attribute: System.Reflection.AssemblyMetadataAttribute
attribute: System.Reflection.AssemblyProductAttribute
attribute: System.Reflection.AssemblyProductAttribute
attribute: System.Reflection.AssemblyTitleAttribute
attribute: System.Reflection.AssemblyTitleAttribute
attribute: System.Runtime.CompilerServices.CompilationRelaxationsAttribute
attribute: System.Runtime.CompilerServices.CompilationRelaxationsAttribute
attribute: System.Runtime.CompilerServices.ExtensionAttribute
attribute: System.Runtime.CompilerServices.ExtensionAttribute
attribute: System.Runtime.CompilerServices.RuntimeCompatibilityAttribute
attribute: System.Runtime.CompilerServices.RuntimeCompatibilityAttribute

Definitely some duplicates, let me know if you want me to run other code. Dumping the properties on the attributes using reflection gives all the same values on the duplicates, i.e. 

value: AssemblyTitleAttribute, Title: System.Private.CoreLib
or
AssemblyProductAttribute Product: Microsoft® .NET Framework
```

This is on Windows 14393.639 and VC 2017 RC 4+26206. No idea what could be causing this, but .Net Native has way too many issues, this is the third unique one in 3 days.

[maxvella]

My conf is

Microsoft Visual Studio Community 2015
Version 14.0.25431.01 Update 3
Win 14393.639

The problem would seem to start from here:

https://github.com/SyntaxC4-MSFT/AutoRest/blob/master/Microsoft.Rest/ClientRuntime/ServiceClient.cs

    protected void InitializeHttpClient(HttpClientHandler httpClientHandler, params DelegatingHandler[] handlers)

    {

        HttpClientHandler = httpClientHandler;

        DelegatingHandler currentHandler = new RetryDelegatingHandler();

        currentHandler.InnerHandler = HttpClientHandler;



        if (handlers != null)

        {

            for (int i = handlers.Length - 1; i >= 0; --i)

            {

                DelegatingHandler handler = handlers[i];

                // Non-delegating handlers are ignored since we always 

                // have RetryDelegatingHandler as the outer-most handler

                while (handler.InnerHandler is DelegatingHandler)

                {

                    handler = handler.InnerHandler as DelegatingHandler;

                }

                handler.InnerHandler = currentHandler;

                currentHandler = handlers[i];

            }

        }



        var newClient = new HttpClient(currentHandler, false);

        FirstMessageHandler = currentHandler;

        HttpClient = newClient;

        Type type = this.GetType();

        HttpClient.DefaultRequestHeaders.UserAgent.Add(new ProductInfoHeaderValue(type.FullName,

            GetAssemblyVersion()));

    }

[SanderMeester]

I came here because my KeyVaultKeyResolver threw a System.Reflection.AmbiguousMatchException - Just in case someone's googling :)
The solution for me was to throw away the Application Insight Extension from my Azure Web App (no need for the extension - the lib is referenced in code)

[SergeyKanzhelev]

@SanderMeester I'm glad you found a solution quickly. I'm sorry for an inconvenience. Just FYI - simply update to the latest version of extension would have been fixed it. Application Insights extension enables some advanced scenarios and does better data collection. For instance, full SQL query will be collected, not just stored procedure names and cross component correlation over http will work.

[morganbr]

I've confirmed the bug in .NET Native and we're tracking it for a future release. I can think of two workarounds:

1. In AutoRest, you can use one of the other GetCustomAttribute or AttributeData methods on Assembly that don't take a Type and do your own filtering that doesn't mind duplicates.
2. For folks trying to get their app to work before 1 can happen, you can add the following to a PropertyGroup in your csproj
   <UseDotNetNativeSharedAssemblyFrameworkPackage>false</UseDotNetNativeSharedAssemblyFrameworkPackage>
   This will disable the shared framework feature. It removes the attribute duplication, but may make your compiled app a few MB larger.

/CC @michalstrehovsky

[jsgoupil]

I was facing the same problem on one slot on Azure... Finally figured it out by updating or deleting the Application Insights as mentioned by @SergeyKanzhelev
Then restart the web app.

here:

[SergeyKanzhelev]

@jsgoupil thanks for update, I hope you found this issue quickly. Sorry for inconvenience.

[bordnerm]

@SergeyKanzhelev You mentioned on Dec 14 (in https://github.com/Microsoft/ApplicationInsights-dotnet-server/issues/241) that an update for the Status Monitor is pending. Do you know the status of that update? What version of the Status Monitor would contain this fix?

We're running into this same issue where we get the System.Reflection.AmbiguousMatchException when trying to use Azure Search; and this error goes away if we uninstall the App Insights Status Monitor.

Thank you!

[bordnerm]

Follow-up to previous post...when I installed the latest App Insights Status Monitor from https://docs.microsoft.com/en-us/azure/application-insights/app-insights-monitor-performance-live-website-now (look for the "Status Monitor installer" link), then I no longer get the AmbiguousMatchException when doing a search using Azure Search.

Note: when looking at Control Panel -> Programs and Features, the version of Status Monitor will be 9.1.0.0.

Thanks!

[rbonestell]

@SergeyKanzhelev
FYI, this is also blocking the use of SQL Server Always Encrypted with Azure Key Vault using Microsoft.Rest.ClientRuntime v2.3.7

I removed the app insights status monitor to resolve the problem.

[SergeyKanzhelev]

@rbonestell thanks for letting us know. You can always update to the latest version of the Status Monitor instead of uninstalling. The issue was fixed

[rbonestell]

@SergeyKanzhelev I tried this first, installing the latest from the website then tried from the web platform installer, both resulted in v9.0.0 being installed on my servers and causing exceptions with my Always Encrypted software.

[SergeyKanzhelev]

@rbonestell if it's not too much to ask - do you have a small repro app? Or perhaps you can get the full exception information including the call stack, the name of custom attribute that was ambiguous and versions of all modules loaded into the process?

[rbonestell]

I don't have a public repo or any more details, but the info below lead me to this GitHub issue which helped me resolve the problem by uninstalling the Application Insights agent from all of our servers.

Using these libraries:
Microsoft.Rest.ClientRuntime v2.3.7
Microsoft.Azure.KeyVault v2.0.6
Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider v2.1.0

Here's the stack trace from my AKV initialization call:

2017-05-06 18:04:32,859 [ERROR] WebApiApplication - Failed to initialize Azure Key Vault provider System.Reflection.AmbiguousMatchException: Multiple custom attributes of the same type found. at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit) at Microsoft.Rest.ServiceClient'1.get_FrameworkVersion() at Microsoft.Rest.ServiceClient'1.get_DefaultUserAgentInfoList() at Microsoft.Rest.ServiceClient'1.SetUserAgent(String productName, String version) at Microsoft.Rest.ServiceClient'1.InitializeHttpClient(HttpClient httpClient, HttpClientHandler httpClientHandler, DelegatingHandler[] handlers) at Microsoft.Azure.KeyVault.KeyVaultClient..ctor(ServiceClientCredentials credentials, DelegatingHandler[] handlers) at Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.SqlColumnEncryptionAzureKeyVaultProvider..ctor(AuthenticationCallback authenticationCallback, String[] trustedEndPoints) at Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.SqlColumnEncryptionAzureKeyVaultProvider..ctor(AuthenticationCallback authenticationCallback)

[rbonestell]

For reference, this is how I initialize the SQL AlwaysEncrypted provider for AKV:
https://github.com/Microsoft/azure-docs/blob/master/articles/sql-database/sql-database-always-encrypted-azure-key-vault.md#register-the-azure-key-vault-provider

[SergeyKanzhelev]

I appreciate all the additional details. @rbonestell I confirmed it is exactly the same problem. I'm very surprised that the Status Monitor update didn't help. I'll investigate whether update of Status Monitor may have some files kept on the system.

Is it the web application you are hosting this code? IIS based? Have you restarted the IIS after updating of Status Monitor?

[rbonestell]

I restarted each server after upgrading, still experienced the same issue. FWIW they are all Azure VMs.

[SergeyKanzhelev]

I double checked Status Monitor. This bug is definitely fixed in v 9.1.0.0. Direct download link. This version will be installed by platform installer. I'm not sure why it wasn't picked up in your environment.

Versions of MicrosoftInstrumentationEngine_x64.dll under C:\Program Files\Microsoft Application Insights\Runtime Instrumentation Agent\x64 should be 0.1.22.16258 or above. Assembly Microsoft.Diagnostics.Instrumentation.Extensions.Base.dll in the same folder should NOT have AssemblyFileVersionAttribute attribute.

[vdevappa]

Is this issue fixed for those unfortunate ones (like me) who use the SDK and cloud services?

[SergeyKanzhelev]

@vdevappa, yes. Windows Azure Diagnostics on cloud services will install the correct version of Status Monitor. Please let me know if you still experience the issue

[vdevappa]

Thanks, that explains why the issues are resolved for me now.

[SvenAelterman]

FYI: I have an Azure Web App that uses Key Vault quite extensively and with updating to the latest App Insights SDK (coming from a v1) this issue popped up. I then upgraded the App Insights Extension to the latest (2.3.2) but no avail. I uninstalled the extension and restarted the app and the app works. I reinstalled the extension, restarted the app again and the app is now still working.
So apparently upgrading the Extension does not suffice, it must be removed and reinstalled.

This is happening on all 3 instances of my app. No issues locally on Win10 dev workstations.
Following Application Insights related NuGet packages are referenced in the app:

• Microsoft.ApplicationInsights 2.3.0
• Microsoft.ApplicationInsights.Agent.Intercept 2.0.7
• Microsoft.ApplicationInsights.DependencyCollector 2.3.0
• Microsoft.ApplicationInsights.PerfCounterCollector 2.3.0
• Microsoft.ApplicationInsights.TraceListener 2.2.0
• Microsoft.ApplicationInsights.Web 2.3.0
• Microsoft.ApplicationInsights.WindowsServer 2.3.0
• Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel 2.3.0

The app targets .NET Framework 4.6.1.

[SergeyKanzhelev]

@iusafaro why update may behave the way @SvenAelterman reported? Is it a bug or special case?

[roboffer]

We are currently seeing this issue in Azure on a site which does use Application Insights. Originally we enabled this in the portal but more recently have bundled it in with our project after the monitoring stopped working.

We experienced the error suddenly without warning when our web app was switched instances in the early hours of the morning yesterday.

I was able to mitigate it by redeploying the same code base to our staging deployment slot and then swapping that with live. That means we do have an environment up on staging that the error still is persisted in if that helps to diagnose anything.

This is initiated by Mongo in our dependency injection code where it attempts to get the framework description.

[AmbiguousMatchException: Multiple custom attributes of the same type found.]
   System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit) +110
   System.Runtime.InteropServices.RuntimeInformation.get_FrameworkDescription() +87
   MongoDB.Driver.Core.Connections.ClientDocumentHelper.GetPlatformString() +9

I have a support case open with Azure support at the moment.

[SergeyKanzhelev]

@roboffer what version of Application Insights extension do you have installed?

@iusafaro @WilliamXieMSFT I wonder can this be a regression with the recently shipped extension version?

[puneetg1983]

I am also engaged on issue for Azure WebApps where things are failing for a particular slot only and they are using MongoDb driver (looks like it is @roboffer only )

We collected a memory dump on this exception and I see the following DLL's and versions loaded

Image name: Microsoft.ApplicationInsights.NLogTarget.dll
Product version:  2.4.1.441
Image name: Microsoft.ApplicationInsights.dll
Product version:  2.4.0.32153
Image name: 2.4.0.0.Microsoft.ApplicationInsights.Extensions.Intercept_x86.dll
Product version:  2.4.0.40021
Image name: Microsoft.ApplicationInsights.Extensions.Base_x86.dll
Product version:  2.0.0.205
Image name: Microsoft.ApplicationInsights.ExtensionsHost.dll
Product version:  2.0.0.205

Does this help ?

[wiktork]

@puneetg1983 For the app service scenario, the newest dll's should be Microsoft.InstrumentationEngine.Extensions.Base_x86.dll and Microsoft.InstrumentationEngine.ExtensionsHost_x64. (Note the name changes). Both should be version 1.0.3+. Can you update to the latest version of ApplicationInsights on this slot?

[roboffer]

@SergeyKanzhelev AFAIK we are using the latest application insights versions, they are installed through NuGet and it shows the versions that are installed are as follows:

Microsoft.ApplicationInsights v2.4.0
Microsoft.ApplicationInsights.Agent.Intercept v2.4.0
Microsoft.ApplicationInsights.DependencyCollector v2.4.1
Microsoft.ApplicationInsights.NLogTarget v2.4.1
Microsoft.ApplicationInsights.PerfCounterCollector v2.4.1
Microsoft.ApplicationInsights.Web v2.4.1
Microsoft.ApplicationInsights.WindowsServer v2.4.1
Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel v2.4.0

What I did just notice is that our staging environment did have Application Insights installed on it as a Site Extension through Kudu and live didn't (however this problem originally happened on live before swapping the slots to resolve). The installed version was 2.0.0, removing this entirely or updating it to the latest version 2.4.7 made no difference and the problem still persists.

After this update I have found DLLs matching those outlined by @wiktork (Microsoft.InstrumentationEngine.ExtensionsHost_x64 v1.3.3.37395 and Microsoft.InstrumentationEngine.Extensions.Base_x86 v1.3.3.37395) in the SiteExtensions folder. I can't confirm what versions were there before.

Without knowing the full details it seems that some code is asking for attributes from a class that has multiple of the same custom attribute. Is the fix to handle that error in the code that is asking for the attributes or is the fix to remove the additional attribute?

[SergeyKanzhelev]

@roboffer Application Insights consists of two parts. One is SDK, another one is additional code instrumentation that is enabled by site extension. That additional code instrumentation makes possible to improve data collection in runtime. Some details in my blog post.

The problem is that older version of that extension by mistake added extra attributes to assembly metadata. This caused problems with AutoRest library as that library expects certain attributes to be defined exactly once.

Please confirm that the issue got fixed after upgrade.

[roboffer]

@SergeyKanzhelev Thanks, that makes sense with what I thought might be happening (changes were being made to a different assembly). Do you know exactly which assembly would have been modified?

We are still seeing the issue after upgrading, but if the assembly that was modified was a system DLL then I'm assuming upgrading won't remove the invalid attribute?

[SergeyKanzhelev]

@roboffer attributes only inserted in runtime in-memory. Not on disk. So upgrade of extension should fix the issue. Have you restarted app after an upgrade of extension?

[KingEgonSANCTITY]

@SergeyKanzhelev I've run into this issue today while deploying to a production web app. In the staging environment I do not get this error, this environment doesn't have AI extension installed. I tried updating the AI web extension but that didn't fix the issue. This does seem to be associated with the mongo driver. Any suggestions are appreciated as this is supposed to be public tomorrow.

[AmbiguousMatchException: Multiple custom attributes of the same type found.]
   System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit) +119
   System.Runtime.InteropServices.RuntimeInformation.get_FrameworkDescription() +127
   System.Lazy`1.CreateValue() +730
   System.Lazy`1.LazyInitValue() +438
   MongoDB.Driver.Core.Connections.ClientDocumentHelper.CreateClientDocument(String applicationName) +76
   MongoDB.Driver.Core.Connections.BinaryConnectionFactory..ctor(ConnectionSettings settings, IStreamFactory streamFactory, IEventSubscriber eventSubscriber) +176
   MongoDB.Driver.Core.Configuration.ClusterBuilder.BuildCluster() +172
   MongoDB.Driver.ClusterRegistry.CreateCluster(ClusterKey clusterKey) +486
   MongoDB.Driver.ClusterRegistry.GetOrCreateCluster(ClusterKey clusterKey) +113
   MongoDB.Driver.MongoClient..ctor(MongoClientSettings settings) +118
   SitefinityWebApp.Mvc.Models.ContextModels.SurveyContext..cctor() +103

[TypeInitializationException: The type initializer for 'SitefinityWebApp.Mvc.Models.ContextModels.SurveyContext' threw an exception.]
   SitefinityWebApp.Mvc.Models.ContextModels.SurveyContext..ctor() +0
   SitefinityWebApp.Mvc.Controllers.EntreeFavoriteSurveyController..cctor() +44

[TypeInitializationException: The type initializer for 'SitefinityWebApp.Mvc.Controllers.EntreeFavoriteSurveyController' threw an exception.]
   SitefinityWebApp.Mvc.Controllers.EntreeFavoriteSurveyController.GetTotals() +36
   SitefinityWebApp.Mvc.Controllers.EntreeFavoriteSurveyController.Insert(EntreeFavoriteModel survey) +92
   lambda_method(Closure , ControllerBase , Object[] ) +175
   System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters) +209
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +35
   System.Web.Mvc.<>c__DisplayClass15.<InvokeActionMethodWithFilters>b__12() +80
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +452
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +452
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +452
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +452
   System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName) +906
   Telerik.Sitefinity.Mvc.ControllerWrapper.Execute() +217
   Telerik.Sitefinity.Mvc.ControllerActionInvoker.ExecuteController(MvcProxyBase proxyControl) +227
   Telerik.Sitefinity.Mvc.ControllerActionInvoker.TryInvokeAction(MvcProxyBase proxyControl, String& output) +311
   Telerik.Sitefinity.Mvc.Proxy.MvcControllerProxy.ExecuteController() +66
   System.Web.UI.Control.PreRenderRecursiveInternal() +200
   System.Web.UI.Control.PreRenderRecursiveInternal() +297
   System.Web.UI.Control.PreRenderRecursiveInternal() +297
   System.Web.UI.Control.PreRenderRecursiveInternal() +297
   System.Web.UI.Control.PreRenderRecursiveInternal() +297
   System.Web.UI.Control.PreRenderRecursiveInternal() +297
   System.Web.UI.Control.PreRenderRecursiveInternal() +297
   System.Web.UI.Control.PreRenderRecursiveInternal() +297
   System.Web.UI.Control.PreRenderRecursiveInternal() +297
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +7487

[SergeyKanzhelev]

@wiktork can it be that extension didn't clean up older binaries after update?

[wiktork]

@jblackwell-thuzi would you be able to provide the list of files in your D:homeSiteExtensionsMicrosoft.ApplicationInsights.AzureWebSitesInstrumentation32? (Instrumentation64 if this is a 64-bit web app)

[KingEgonSANCTITY]

@wiktork Yes, I can provide both. My app is 64-bit. Thanks let me know if you need anything else.

32-bit

Mode                LastWriteTime         Length Name                          
----                -------------         ------ ----                          
-a----         1/3/2018   4:05 PM          29512 Microsoft.Diagnostics.Instrumentation.Extensions.Base.dll   
-a----         1/3/2018   4:05 PM         554304 Microsoft.InstrumentationEngine.Extensions.Base_x86.dll     
-a----         1/3/2018   4:05 PM            486 Microsoft.InstrumentationEngine.Extensions.config           
-a----         1/3/2018   4:05 PM         259392 Microsoft.InstrumentationEngine.ExtensionsHost_x86.dll      
-a----         1/3/2018   4:05 PM        1105648 MicrosoftInstrumentationEngine_x86.dll                      
-a----         1/3/2018   4:05 PM            448 ProductionBreakpoints_x86.config                            
-a----         1/3/2018   4:05 PM         688848 ProductionBreakpoints_x86.dll 
-a----         1/3/2018   4:05 PM         288952 SnapshotHolder_x86.exe 

64-bit

-a----         1/3/2018   4:05 PM          29512 Microsoft.Diagnostics.Instrumentation.Extensions.Base.dll   
-a----         1/3/2018   4:05 PM         718656 Microsoft.InstrumentationEngine.Extensions.Base_x64.dll     
-a----         1/3/2018   4:05 PM            486 Microsoft.InstrumentationEngine.Extensions.config           
-a----         1/3/2018   4:05 PM         312120 Microsoft.InstrumentationEngine.ExtensionsHost_x64.dll      
-a----         1/3/2018   4:06 PM        1283832 MicrosoftInstrumentationEngine_x64.dll                      
-a----         1/3/2018   4:05 PM            448 ProductionBreakpoints_x64.config                            
-a----         1/3/2018   4:05 PM         790224 ProductionBreakpoints_x64.dll 
-a----         1/3/2018   4:05 PM         341176 SnapshotHolder_x64.exe

[puneetg1983]

Probably a good idea to run the following command from KUDU -> Debug Console -> Powershell as that will show the file versions easily...

dir *.dll | fl

[KingEgonSANCTITY]

@wiktork The versions for the 64-bit.
Thanks @puneetg1983.

Name           : Microsoft.Diagnostics.Instrumentation.Extensions.Base.dll
Length         : 29512
CreationTime   : 1/3/2018 4:06:02 PM
LastWriteTime  : 1/3/2018 4:05:59 PM
LastAccessTime : 1/3/2018 4:06:02 PM
Mode           : -a----
LinkType       : 
Target         : 
VersionInfo    : File:             D:\home\SiteExtensions\Microsoft.Application
                 Insights.AzureWebSites\Instrumentation64\Microsoft.Diagnostics
                 .Instrumentation.Extensions.Base.dll
                 InternalName:     
                 Microsoft.Diagnostics.Instrumentation.Extensions.Base.dll
                 OriginalFilename: 
                 Microsoft.Diagnostics.Instrumentation.Extensions.Base.dll
                 FileVersion:      0.0.0.0
                 FileDescription:   
                 Product:          
                 ProductVersion:   0.0.0.0
                 Debug:            False
                 Patched:          False
                 PreRelease:       False
                 PrivateBuild:     False
                 SpecialBuild:     False
                 Language:         Language Neutral


Name           : Microsoft.InstrumentationEngine.Extensions.Base_x64.dll
Length         : 718656
CreationTime   : 1/3/2018 4:06:02 PM
LastWriteTime  : 1/3/2018 4:05:59 PM
LastAccessTime : 1/3/2018 4:06:02 PM
Mode           : -a----
LinkType       : 
Target         : 
VersionInfo    : File:             D:\home\SiteExtensions\Microsoft.Application
                 Insights.AzureWebSites\Instrumentation64\Microsoft.Instrumenta
                 tionEngine.Extensions.Base_x64.dll
                 InternalName:     InstrumentationEngine.Extensions.Base
                 OriginalFilename: 
                 FileVersion:      1.0.3.37395
                 FileDescription:  
                 Product:          Microsoft® Visual Studio®
                 ProductVersion:   1.0.3.37395
                 Debug:            False
                 Patched:          False
                 PreRelease:       False
                 PrivateBuild:     False
                 SpecialBuild:     False
                 Language:         English (United States)


Name           : Microsoft.InstrumentationEngine.ExtensionsHost_x64.dll
Length         : 312120
CreationTime   : 1/3/2018 4:06:02 PM
LastWriteTime  : 1/3/2018 4:05:59 PM
LastAccessTime : 1/3/2018 4:06:02 PM
Mode           : -a----
LinkType       : 
Target         : 
VersionInfo    : File:             D:\home\SiteExtensions\Microsoft.Application
                 Insights.AzureWebSites\Instrumentation64\Microsoft.Instrumenta
                 tionEngine.ExtensionsHost_x64.dll
                 InternalName:     InstrumentationEngine.ExtensionsHost
                 OriginalFilename: 
                 FileVersion:      1.0.3.37395
                 FileDescription:  
                 Product:          Microsoft® Visual Studio®
                 ProductVersion:   1.0.3.37395
                 Debug:            False
                 Patched:          False
                 PreRelease:       False
                 PrivateBuild:     False
                 SpecialBuild:     False
                 Language:         English (United States)


Name           : MicrosoftInstrumentationEngine_x64.dll
Length         : 1283832
CreationTime   : 1/3/2018 4:06:02 PM
LastWriteTime  : 1/3/2018 4:06:00 PM
LastAccessTime : 1/3/2018 4:06:02 PM
Mode           : -a----
LinkType       : 
Target         : 
VersionInfo    : File:             D:\home\SiteExtensions\Microsoft.Application
                 Insights.AzureWebSites\Instrumentation64\MicrosoftInstrumentat
                 ionEngine_x64.dll
                 InternalName:     MicrosoftInstrumentationEngine_x64
                 OriginalFilename: MicrosoftInstrumentationEngine_x64.dll
                 FileVersion:      1.0.3.37395
                 FileDescription:  Microsoft Instrumentation Engine
                 Product:          Microsoft® Visual Studio®
                 ProductVersion:   1.0.3.37395
                 Debug:            False
                 Patched:          False
                 PreRelease:       False
                 PrivateBuild:     False
                 SpecialBuild:     False
                 Language:         English (United States)


Name           : ProductionBreakpoints_x64.dll
Length         : 790224
CreationTime   : 1/3/2018 4:05:36 PM
LastWriteTime  : 1/3/2018 4:05:36 PM
LastAccessTime : 1/3/2018 4:05:36 PM
Mode           : -a----
LinkType       : 
Target         : 
VersionInfo    : File:             D:\home\SiteExtensions\Microsoft.Application
                 Insights.AzureWebSites\Instrumentation64\ProductionBreakpoints
                 _x64.dll
                 InternalName:     ProductionBreakpoints_x64
                 OriginalFilename: ProductionBreakpoints_x64.dll
                 FileVersion:      2017121202 
                 79af0959ba3685f5c477036da0eb8842efd0a0d5 
                 ProductionBreakpoints-Release
                 FileDescription:  
                 Product:          Microsoft® Visual Studio®
                 ProductVersion:   2017121202 
                 79af0959ba3685f5c477036da0eb8842efd0a0d5 
                 ProductionBreakpoints-Release
                 Debug:            False
                 Patched:          False
                 PreRelease:       False
                 PrivateBuild:     False
                 SpecialBuild:     False
                 Language:         English (United States)

[wiktork]

Thanks for putting this together. Unfortunately these look like the latest bits. I will try repro this issue, but so far I've had no luck.

[wiktork]

Also @jblackwell-thuzi @roboffer if you guys are willing to share D:homeLogFileseventlog.xml, we do record some instrumentation failures into this log with the newest version of appinsights. Note there may be unrelated events there too, so please be sure to look over the file first.

[KingEgonSANCTITY]

@wiktork Thanks for looking into this. If it helps I was at version 2.00 of the web extension when I encountered the issue. Also it seems kind of a random occurrence as I have been using the mongo driver for a while now in the instance that now has the error. All instances of the driver use cause this now whereas this was not the case in as little as a month ago.

[roboffer]

@SergeyKanzhelev It looks like updating the application insights extension has solved our issue, originally I was using the "Restart Site" button in the Site Extensions part of Kudu after making changes and the problem still persisted even after removing application insights entirely.

Killing the w3wp exe from Process Explorer seems to have made it work so it does look like it was v2.0.0 as mentioned above. From reading the extension moves with the slot, so this would explain why switching from staging to live resolved it. I will verify this on our next deployment.

@wiktork I am happy to share that file with you privately

[roboffer]

@jblackwell-thuzi Try updating the application insights extension version and then killing the w3wp as I mentioned above, that solved it for us

[MattHartz]

Hi everyone,

So to address this issue do we just need to disable or update application insights?

Thanks!

[wiktork]

@MattHartz Yes, but make sure you restart all your instances to ensure that the updated version has taken effect. As mentioned by @roboffer, you may need to terminate your w3wp.exe processes from Process Explorer.

[MiYanni]

Seems the issue was resolved. Closing.