Azure-cosmos-dotnet-v2: AggregateException (Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll') in ASP.NET Web API project

[andresjacobo]

Similar to Issue #118.

I am running an ASP.NET project in .NET 4.6 using the latest 1.11.3 DocumentDB SDK and whenever I try to do any query using CreateDocumentQuery, the query is created successfully but trying to iterate through the results throws an AggregateException, making DocumentDB useless since queries can't be performed.

I've verified that Microsoft.Azure.Documents.ServiceInterop.dll is indeed in the project's bin folder.

What's interesting (and frustrating!) is that in a regular .NET console application using the same .NET SDK version and the same DocumentDB SDK version, the queries are executed without a problem, leading me to think the problem is specific to ASP.NET.

Please help.

Below is a stack trace of the exception thrown:

at Microsoft.Azure.Documents.ServiceInteropWrapper.CreateServiceProvider(String configJsonString, IntPtr& serviceProvider)\r\n at Microsoft.Azure.Documents.Query.QueryPartitionProvider.InitializeServiceProvider()\r\n at Microsoft.Azure.Documents.Query.QueryPartitionProvider.GetPartitionedQueryExecutionInfo(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition)\r\n at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.<GetPartitionedQueryExecutionInfoAsync>d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextFactory.<CreateDocumentQueryExecutionContextAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Documents.Linq.DocumentQuery1.d__11.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Documents.Linq.DocumentQuery1.<ExecuteAllAsync>d__1b.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Documents.Linq.DocumentQuery1.d__d.MoveNext()`

[mingaliu]

What is the exact exception? Was it looks like:
System.DllNotFoundException: Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

If that is the case, you can try workaround in http://stackoverflow.com/questions/344608/unmanaged-dlls-fail-to-load-on-asp-net-server/4598747.

[andresjacobo]

@mingaliu Thanks for the quick response. You are correct, the exact exception is:

InnerException {"Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)"} System.Exception {System.DllNotFoundException}

I will try the workaround you mentioned and report back. Since our project is deployed to Azure from Visual Studio on my local machine, I wonder if the deployed version will have any problems?

[andresjacobo]

The workaround suggested works on my local machine. I wonder if it will work once deployed in production?

[rnagpal]

@andresjacobo Does your ASP.NET Web project directly references DocumentDB Nuget package or is it through a second level dependency? I mean, is it like: ASP.NET Web project referencing a ASP.NET class library which references DocumentDB Nuget package.

If it's second level, try to directly reference the DocumentDB Nuget package and ensure that
your ASP.NET Web project gets the following import:

Let me know if you still see the issue.

[andresjacobo]

@rnagpal To use DocumentDB on our ASP.NET Web project, I simply used Nuget inside Visual Studio to download the DocumentDB SDK. This, I believe, automatically adds a direct reference in the project's 'references' to Microsoft.Azure.Documents.Client.

So yes, I would say that even with a direct reference, the problem occurs. What can it be?

[rnagpal]

@andresjacobo Apart from the assembly reference to Microsoft.Azure.Documents.Client.dll ensure that you also get the following import:
<Import Project="..\packages\Microsoft.Azure.DocumentDB.1.11.3\build\Microsoft.Azure.DocumentDB.targets" Condition="Exists('..\packages\Microsoft.Azure.DocumentDB.1.11.3\build\Microsoft.Azure.DocumentDB.targets')" />
If you have it(which I believe), then try pasting the following in your project:
<Content Include="$(NugetPack)\Microsoft.Azure.DocumentDB.1.11.3\runtimes\win7-x64\native\DocumentDB.Spatial.Sql.dll"> <Link>DocumentDB.Spatial.Sql.dll</Link> <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory> </Content> <Content Include="$(NugetPack)\Microsoft.Azure.DocumentDB.1.11.3\runtimes\win7-x64\native\Microsoft.Azure.Documents.ServiceInterop.dll"> <Link>Microsoft.Azure.Documents.ServiceInterop.dll</Link> <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory> </Content>

Let me know if that helps.

[andresjacobo]

@rnagpal I do indeed have the first import. I then linked the other two libraries but when I ran the project I got the same exception (DLLNotFound).

Is there a particular place I should paste the content you mentioned? I don't think it should make a difference, but I can't get it to work.

[PratikChavda]

@rnagpal
I am getting same error but scenario is different
.net SDK version : 1.11.3
.Net version : 4.5.2
.Net Project Type : .net Class library.

I am refering this class library in unit test project.
I am getting this error when code performs await query.ExecuteNextAsync().
I have two sets of similar test

1. for single partition and it execute successfully
2. for two partition and it fails every time.

Below is stack trace:

Result StackTrace:
at Microsoft.Azure.Documents.ServiceInteropWrapper.CreateServiceProvider(String configJsonString, IntPtr& serviceProvider)
at Microsoft.Azure.Documents.Query.QueryPartitionProvider.Initialize()
at Microsoft.Azure.Documents.Query.QueryPartitionProvider.GetPartitionedQueryExecutionInfoInternal(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected)
at Microsoft.Azure.Documents.Routing.PartitionRoutingHelper.GetProvidedPartitionKeyRanges(SqlQuerySpec querySpec, Boolean enableCrossPartitionQuery, Boolean parallelizeCrossPartitionQuery, PartitionKeyDefinition partitionKeyDefinition, QueryPartitionProvider queryPartitionProvider, String clientApiVersion, QueryInfo& queryInfo)
at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.BackoffRetryUtility1.<>c__DisplayClass2.<<ExecuteAsync>b__0>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Documents.BackoffRetryUtility1.d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Documents.BackoffRetryUtility1.<ExecuteRetry>d__1b.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Documents.BackoffRetryUtility1.d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.Linq.DocumentQuery1.<ExecuteNextPrivateAsync>d__181.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at MyApp.Data.DocumentDB.Azure.DocumentRepository1.d__38.MoveNext() in C:\Users\username\Source\Repos\MyApp\MyApp.Data.DocumentDB\Azure\DocumentRepository.cs:line 67
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at MyApp.Data.DocumentDB.Tests.DocumentRepositoryTest.d__5.MoveNext() in C:\Users\username\Source\Repos\MyApp\MyApp.Data.DocumentDB.Tests\DocumentRepositoryTest.cs:line 34
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
Result Message:
Test method MyApp.Data.DocumentDB.Tests.DocumentRepositoryTest.QueryAsync threw exception:
System.DllNotFoundException: Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

[rnagpal]

@PratikChavda The issue in the first portion of your Stack trace has been fixed in 1.11.4 release. Please upgrade and let me know if you get DllNotFound exception. I'm investigating the issue reported by @andresjacobo for the ASP.Net project.

[rnagpal]

@andresjacobo As long as it's in an ItemGroup tag under top level Project tag, you should be good.

Can you confirm that you have both the native dlls: Microsoft.Azure.Documents.ServiceInterop.dll and DocumentDB.Spatial.Sql.dll in the bin dir? I've seen reports wherein if the Spatial dll is not there at the same place as ServiceInterop, you get this exception as ServiceInterop depends upon Spatial dll and both need to be present. Unfortunately, I haven't been able to repro this on my end using an ASP.NET Web app hosted on azure but given some folks have reported it to us, I believe there is some configuration missing that causes this behavior.

Can you also try the workarounds mentioned in https://blogs.msdn.microsoft.com/jorman/2007/08/31/loading-c-assemblies-in-asp-net/ specifically:

`In web.config, add the following…use a path where the Native C++ DLL is located:

In global.asax, add the following:

protected void Application_Start(object sender, EventArgs e){
String _path = String.Concat(System.Environment.GetEnvironmentVariable("PATH"), ";", ConfigurationSettings.AppSettings["NativePath"]);
System.Environment.SetEnvironmentVariable("PATH", _path, EnvironmentVariableTarget.Process);
}
`
Let me know if that helps!

[rnagpal]

Just to have this post contain all possible cases when you run into this, if you are seeing this exception from a VS Unit test project trying to test an ASP.NET class library project that has a reference to DocumentDB Nuget package, you may have to add the following as an attribute to the test class:
[DeploymentItem(@"Microsoft.Azure.Documents.ServiceInterop.dll")] //force copy of dependent dll. [DeploymentItem(@"DocumentDB.Spatial.Sql.dll")] //force copy of dependent dll.

[andresjacobo]

@rnagpal Thanks for the info. I can confirm both DLLs you mention are in the bin dir of the ASP.NET project.

I've tried every combination of all the advice you've given me, and this is what worked in the end:

I added the following code to the global.asax file:
String _path = String.Concat(System.Environment.GetEnvironmentVariable("PATH"), ";", System.AppDomain.CurrentDomain.RelativeSearchPath); System.Environment.SetEnvironmentVariable("PATH", _path, EnvironmentVariableTarget.Process);

With this in place, the DLLs seem to load fine and queries don't throw an AggregateException. This seems like a fairly good solution given that a) I don't pollute the System32 directory b) There's no version specific path that would break when the DocumentDB SDK is updated.

My only remaining question is why would we need to do this? Surely there is something specific in my project that causes this to happen. Given how often DocumentDB is used inside ASP.NET, I wouldn't think this is a widespread issue.

[rnagpal]

@andresjacobo Glad to hear that. I already reported this on our ASP.NET Discussions alias but haven't got any response yet. I'll update you when I hear from them.

This is not specific to DocumentDB as other packages with native dlls have the same issue. Also since I'm unable to repro this on my end, I don't believe this is a widespread issue as we would have heard this loud enough.

Do you mind creating a small repro app for me and attaching that to my email(listed in my profile) and I'll see if I can repro it? Just run this query on click of a button to repro this. Keep it as simple as possible. I can follow up with you offline in case I've any queries on the sample app.

[andresjacobo]

@rnagpal Sorry it took me some time.

I've created a super simple ASP.NET project in VS and (un) fortunately I was able to reproduce the problem.

To recreate the problem, all I did was:

1. Create a new project.
2. Download DocumentDB SDK from Nuget.
3. Create a test controller using the built in template from VS.
4. Perform a query inside the controller.
5. Exception is thrown :boom:

I've omitted the DB key from the project. If you need to connect to our specific instance, I'd be happy to give it to you in a secure manner.

Let me know if you are able to reproduce the issue.

Download link

[clifflo]

I have faced a weird sitution. I was working for a traditional ASP.NET 4.5.2 Framework (not Core). When I use the DocumentDB in IIS express in Visual Studio it works. After that I publish the whole project to Azure and it still works. However when I use that in local IIS it does not work and have the same exception as the start of this issue. Afterwards I have referred to the workaround solition suggested by @mingaliu and I put the DocumentDB.Spatial.Sql.dlland the Microsoft.Azure.Documents.ServiceInterop.dll into the C:\Windows\System32. It works!!! So I think the problem is due to some dependencies error of the .NET Framework itself, rather than the DocumentDB SDK.

[thuru]

@rnagpal : I think we're still stuck in that. I referenced the new 1.12.0 version of the Document DB NuGet and still having the error.

Could not load file or assembly 'file:///D:\home\site\wwwroot\binDocumentDB.Spatial.Sql.dll' or one of its dependencies. The module was expected to contain an assembly manifest.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.BadImageFormatException: Could not load file or assembly 'file:///D:\home\site\wwwroot\binDocumentDB.Spatial.Sql.dll' or one of its dependencies. The module was expected to contain an assembly manifest.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Assembly Load Trace: The following information can be helpful to determine why the assembly 'file:///D:\home\site\wwwroot\binDocumentDB.Spatial.Sql.dll' could not be loaded.

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

Stack Trace:

[BadImageFormatException: Could not load file or assembly 'file:///D:\home\site\wwwroot\binDocumentDB.Spatial.Sql.dll' or one of its dependencies. The module was expected to contain an assembly manifest.]
System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) +0
System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) +485
System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark) +190
System.Reflection.Assembly.LoadFrom(String assemblyFile) +54
WebActivator.PreApplicationStartCode.Start() +157

[InvalidOperationException: The pre-application start initialization method Start on type WebActivator.PreApplicationStartCode threw an exception with the following error message: Could not load file or assembly 'file:///D:\home\site\wwwroot\binDocumentDB.Spatial.Sql.dll' or one of its dependencies. The module was expected to contain an assembly manifest..]
System.Web.Compilation.BuildManager.InvokePreStartInitMethodsCore(ICollection1 methods, Func1 setHostingEnvironmentCultures) +903
System.Web.Compilation.BuildManager.InvokePreStartInitMethods(ICollection`1 methods) +170
System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded) +169
System.Web.Compilation.BuildManager.ExecutePreAppStart() +172
System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +907

[HttpException (0x80004005): The pre-application start initialization method Start on type WebActivator.PreApplicationStartCode threw an exception with the following error message: Could not load file or assembly 'file:///D:\home\site\wwwroot\binDocumentDB.Spatial.Sql.dll' or one of its dependencies. The module was expected to contain an assembly manifest..]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +539
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +118
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +730

[clifflo]

Hi @thuru with regard to your case I think you should try adding the file to D:\home\site\wwwroot\bin\ or perhaps as of my case, adding it to C:\Windows\System32. In general I would say the problem is somehow related to .NET Framework itself not being able to the load the file in Nuget.

[rnagpal]

@thuru I'll send you an invite for a quick call so that we can close this issue asap as it seems that you have been having this issue from long.

@clifflo Instead of that hack, can you try the following code to the global.asax file:
String _path = String.Concat(System.Environment.GetEnvironmentVariable("PATH"), ";", System.AppDomain.CurrentDomain.RelativeSearchPath); System.Environment.SetEnvironmentVariable("PATH", _path, EnvironmentVariableTarget.Process);

and see if that resolves the issue?

[rnagpal]

@clifflo : The issue you are running into is different from Thuru. You might be seeing DllNotFoundException but Thuru is seeing BadImageFormatException which have different causes.

@thuru I see that this exception is coming from an external package WebActivator that seems to load our native dlls. Since these are native dlls, Assembly.LoadFrom() call is bound to fail with BadImageFormatException. Please refer to this for more details: http://stackoverflow.com/questions/18325799/could-not-load-file-or-assembly-hunspellx64-dll-while-using-nhunspell-nuget-pack

Can you try uninstalling the current version of WebActivator and install the latest one(looks like upgrade is not supported)?

Please keep me updated on this issue.

[wonix]

I got the same exception as the title of this issue indicates.

Here's my setup

• Visual Studio 2017
• .NET standard 4.6.1
• Web API 2.0 using Service Fabric
• The API application references library that internally references _Microsoft.Azure.DcoumentDB v1.13.0_

I have another Service Fabric application which does not use Web API with a similar setup that does not throw the exception. The issue does seem to specific to mixing with Web API.

Directly referencing the same library within the Web API project resolved that issue for me. :+1:

[rnagpal]

@Wonix If your target application is project.json based and you are referring a library project that internally references DocumentDB nuget package, those native dependencies will not be copied as they are based on .targets file which is not supported in project.json projects, so yes you would need to explicitly reference DocumentDB nuget package in your target application which is project.json.

In your case, is Web API 2.0 a project.json based project?

[wonix]

@rnagpal, my application is not project.json based. it was created using the Service Fabric Stateless Service template in Visual Studio 2015. The library project is also not project.json based.

[lehn0058]

I'm having the same issue as well. I have the latest NuGet update (1.13.1). DocumentDB queries work fine an a console application, but when trying to query from a WebAPI controller with the exact same code I get the exception:

System.DllNotFoundException: Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

Note this is only on my local machine - deploying the web service to Azure web apps works fine.

[lehn0058]

I was able to fix this error when running locally by adding the following to my web.config under the system.web section:

<hostingEnvironment shadowCopyBinAssemblies="false" />

[dmunozgaete]

thanks @lehn0058 for the fix, i got the same problem and hours wasted trying to fix!.
your solution work for me!.

[YoungjaeKim]

I almost fight for a day and still unsolved.
WebApi .Net 4.6.1 project throws BadImageFormatException DocumentDB.Spatial.Sql while WPF 4.6.1 works fine.
How can I precisely diagnose and fix this issue?

I already have two dll files (Interop and Spatial) in bin folder correctly. Setup almost all combinations of x64/AnyCPU whatever configurations.

[YoungjaeKim]

Today, I tested with fresh ASP.NET WebApi project.

• Visual Studio 2017 Enterprise
• Azure Api App template (Azure SDK 3.0.1)
• .Net framework 4.6
• DocDB library 1.14.1
• WebActivatorEx 2.2.0
• Newtonsoft.Json 9.0.1

In below table;
StartUp stands for a project that created from Azure Api App template.
Library stands for depended .NET class library (same .Net Framework 4.6) of the Startup project.

Both have same DocDB library and code to read one DocDB record.

|StartUp | Library |Result|
|---------|--------|--------|
|AnyCPU | - | OK|
|x64 | - | Error|
|AnyCPU | AnyCPU | OK|
|AnyCPU | x64| Error|
|x64 | AnyCPU | Error|
|x64 | x64|Error|

In summary, DocumentDB library should be used with AnyCPU configuration.
Build AnyCPU library but running on the x64 does not work.

In addition, I noticed that ALL configurations raise BadImageFormatException from DocumentDB.Spatial.Sql but AnyCPU runs fine if we ignore it.

For me, (1) adding <hostingEnvironment shadowCopyBinAssemblies="false" /> at system.web does NOT help, (2) adding PATH in Global.asax does NOT help either. I didn't try to copy system32 as this is too hacky.

• Reproduce repository: https://github.com/YoungjaeKim/DocumentDBWeb

[rnagpal]

@YoungjaeKim This is most likely an issue with your application trying to load the Spatial and Interop dlls that are native. If your process is trying to load these assemblies using Assembly.LoadFrom API it will fail with BadImageFormatException as expected: https://msdn.microsoft.com/en-us/library/system.badimageformatexception. Can you search for Assembly.LoadFrom in your application and ensure that this doesn't gets called for native assemblies?

Please refer to https://stackoverflow.com/questions/18325799/could-not-load-file-or-assembly-hunspellx64-dll-while-using-nhunspell-nuget-pack in case you are running into this issue.

Also, I see that you are using WebActivator package which might be the culprit. The above link says that the latest version of WebActivator seems to have fixed this issue. Can you double check that as well?

Note that you will run into this issue even if you don't have a single line of DocDB code in your application. This issue happens at startup of your application which tries to load all the assemblies using Assembly.LoadFrom which is the culprit.

Please let me know if that helps and if not we can investigate further.

[jstawski]

I just want to add to the discussion. We received

Unable to find an entry point named 'CreateServiceProvider' in DLL 'Microsoft.Azure.Documents.ServiceInterop.dll'.

Our website is hosted in Azure App Service as Azure Website. When I went to the bin folder I noticed that both of the DLLs mentioned here: Microsoft.Azure.Documents.ServiceInterop.dll and DocumentDB.Spatial.Sql.dll were there twice, but the second one has the temp extension .dll.MSDEPLOYTMP.

I am pretty sure that extension is there because of another problem we were having with deployment and locked files. This thread suggested to add the key MSDEPLOY_RENAME_LOCKED_FILES with a value of 1. Supposedly it tells te deployment to try to rename locked files.

Furthermore, we had redeployed the application and it started working again only to stop working after about an hour or so.

Do you think these renamed files could be causing the problem?

[lehn0058]

While the shadow copy setting worked for most of our Dev's, there were still a few that were getting this error. It turns out, it was a permission issue! We debug locally against IIS (not IIS express) and Application Identity and Network Service did not have valid permissions. Not sure why they needed different permissions to load the Microsoft.Azure.Documents.ServiceInterop.dll, but upping the permissions on the local Dev environments resolved this error completely for us.

[YoungjaeKim]

@lehn0058 good insight. but I'm using the library on Azure WebApp which cannot change permissions :(

[Valchris]

I faced the similar issue on version 1.11.1 using 64 bit process. My console based application was giving an error response of: Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll'. After reading this thread, I uninstalled the nuget package and re-installed the latest version 1.14.1. The re-install added a missing targets reference and the issue was resolved.

[jackbond]

I added a separate issue before finding this one.

https://github.com/Azure/azure-documentdb-dotnet/issues/312

I'm seeing the same error using the core Nuget package, i.e. Microsoft.Azure.DocumentDB.Core The exception we're getting is for "The specified module could not be found"

Some environment info:

Service Fabric
4.7 .NET Framework
64 bit executables

Currently we're using the classic Nuget Microsoft.Azure.DocumentDB, but we want to move to the Core version for its .NETStandard support.

[jonathanfishbein1]

This is still a problem for me. It seems the error is showing up when my code is executed within the Azure functions runtime as the error doesn't happen when I unit test the code?

dotnet --version 2.1.3
Microsoft.Azure.DocumentDB.Core (1.7.1)
"azure-functions-core-tools": "^2.0.1-beta.22"
F#
Azure Functions

Any help will be much appreciated. Thank you

[MovGP0]

I have the same problem with an LinqPad script referencing Microsoft.Azure.DocumentDB.Core version 1.7.1.

[rposener]

Same error here targeting netstandard2.0 with these nugets:

<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.6" /> <PackageReference Include="Sendgrid" Version="9.9.0" /> <PackageReference Include="StrongGrid" Version="0.35.0" /> <PackageReference Include="Microsoft.Azure.DocumentDB.Core" Version="1.6.0" /> <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.7" />

It seems that I can add Microsoft.Azure.DocumentDB v.1.19.1 (along with the .Core version) and it works.

[kirankumarkolli]

@rposener core nuget on standard project failed but fx nuget worked?

[kirankumarkolli]

@ealsur could you please take a look at the azure functions issue mentioned by @jonathanfishbein1?

[rposener]

@kirankumarkolli yes, had to have both but then works targeting netstandard2.0 on windows machine.

[ealsur]

@rposener Are you adding both Functions Sdk versions to the same project? Are you deploying the Function as a pre-compiled function to a Function App in App Service Mode? Is the App Service a Linux version or Windows?

@jonathanfishbein1 Are you also deploying pre-compiled code in the Function App? What Functions Sdk version are you using? Does the deploy include the Interop DLL?

[jonathanfishbein1]

@ealsur Thanks for your response. I am deployed pre-compiled code. I'm using

azure-functions-core-tools": "^2.0.1-beta.22

I have fixed the problems by adding a script that adds
DocumentDB.Spatial.Sql.dll

and

Microsoft.Azure.Documents.ServiceInterop.dll

to the target folder

[steevcoco]

I am seeing the same exception.

"Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)"

It's in an Azure Function App, V2 preview; in Visual Studio (precompiled --- and not yet deployed at all).

The exception is raised at

IDocumentQuery<T>.ExecuteNextAsync<T>()

Neither Microsoft.Azure.Documents.ServiceInterop.dll nor DocumentDB.Spatial.Sql.dll appear in the bin directory --- when the project is compiled.

[kirankumarkolli]

@steevcoco, @rposener could you please share the repro projects for us to troubleshoot this further.

cc: @ausfeldt

[MovGP0]

as a reference, this one is working for me:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <AzureFunctionsVersion>v2</AzureFunctionsVersion>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'">
    <LangVersion>latest</LangVersion>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
    <LangVersion>latest</LangVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.7" />
    <PackageReference Include="Newtonsoft.Json" Version="10.0.3" />
<!--
    <PackageReference Include="..." Version="..." />
-->
  </ItemGroup>
  <ItemGroup>
<!--
    <ProjectReference Include="..." />
-->
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
  <ItemGroup>
    <Folder Include="Properties\PublishProfiles\" />
  </ItemGroup>
</Project>

[steevcoco]

Hello @kirankumarkolli

I cannot easily share the code ... It is in two internal repos. If out of desperation it would be helpful, I could try and arrange something.

I can cure the problem here; and I can explain.

To cure it, I simply need to manually copy the assemblies to my own bin directory:
Microsoft.Azure.Documents.ServiceInterop.dll
DocumentDB.Spatial.Sql.dll

This is a Function Apps v2 App ... (NetStandard 2.0) and it is running locally in Visual Studio. I am using a DocumentClient to make a simple query ... and at IDocumentQuery<T>.ExecuteNextAsync<T>() the exception is raised; because the dll has not been copied. AND IN FACT, this App does not directly reference Microsoft.Azure.DocumentDB.Core. I have a shared library (NetStandard 2.0) that implements a wrapper around DocumentClient, and the Function App references this library. This library is using Microsoft.Azure.DocumentDB.Core ... When built, the platform dlls are not copied.

So on the local machine, if I simply manually copy the above dlls, it works. I cannot publish this App to test it on the server! Perhaps because we can't publish V2 preview Apps? I don't know at this time ... But publishing this Function App fails.

[kwilkins]

I ran into this problem today, I had to manually copy Microsoft.Azure.Documents.Client.dll into the parent project's bin folder. It was in my project that directly referenced DocumentDB, but the other project that consumed it only had Microsoft.Azure.Documents.ServiceInterop.dll and would fail at runtime when looking for Microsoft.Azure.Documents.Client.dll

Hope this gets resolved soon, I love using DocumentDB :)

[MovGP0]

still not working in Microsoft.Azure.DocumentDB.Core version 1.8.1 when using LinqPad.

[hackked]

I'm having the same issue with an Azure Functions v2 app and trying to use the DocumentDB.Core nuget package. The two native DLLs are placed into runtimes\win\native in the bin folder, but when running under Azure Functions host, they are not located. If I copy these two out of the native folder to the root \bin folder the code will work as expected.

I've seen someone mention adding a reference to the Microsoft.Azure.DocumentDB package which does work, but only because that nuget package contains logic to place these same two native DLLs into the root of the \bin folder.

[ausfeldt]

Microsoft.Azure.DocumentDB.Core has been deprecated. Can you try 1.21.0 https://www.nuget.org/packages/Microsoft.Azure.DocumentDB/

[jackbond]

@ausfeldt I tried, but there appears to be a MASSIVE BUG. These may be the steps to reproduce...

1) Add a reference to Microsoft.Azure.DocumentDB 1.21.0 from a .netstandard project (Project X)
2) Add a reference to Project X from another .netstandard project (Project Y)

When you compile, project Y appears to reference Microsoft.Azure.DocumentDb.Core and then you get all sorts of type mismatches in your solution. And to confirm, I cleaned my solution, even deleted everything and did a clean get from TFS, removed every reference to DocumentDb nugets, and did a file system search of DocumentDb.Core. No references at all. Then I added the Microsoft.Azure.DocumentDB Nuget to project X, and all of a sudden, references to Microsoft.Azure.DocumentDB.Core started show up in *.deps.json and projects.assets.json.

I don't know much about how nuget packages are created, but it would appear, in your process to merge the two nuget packages, you've got a reference to the core version in the none core version.

[ausfeldt]

@jackbond Thanks, working on a fix now.

[jackbond]

@ausfeldt Just to confirm, you were able to reproduce the issue?

[ausfeldt]

@jackbond similar issue here https://github.com/Azure/azure-documentdb-dotnet/issues/442. I can reproduce if Project X is .NET Standard and Project Y is .NET Framework. Are you sure both are .NET Standard?

[jackbond]

@ausfeldt My mistake, Project Y is in fact a .NET Framework project.

That being said, I literally cannot believe that this bug made it into a published nuget package. If someone isn't fired for this, then your project manager is OUT OF HIS FUCKING MIND. Do you guys have what is generally known in the rest of the industry as a "QA Team?" If the DocumentDb team is unfamiliar with the concept, it's a group of engineers who TEST software BEFORE it is released.

I'm no expert in nuget packaging, but looking at the contents of the 1.21.0 release, this merged package had ABSOLUTELY ZERO chance of functioning in a solution which mixes .NET standard and .NET framework projects. ABSOLUTELY ZERO. That is beyond unforgiveable.

[ausfeldt]

@jackbond If Project Y is a .NET Framework, then I am able to reproduce it. Thanks.

[ausfeldt]

@jackbond The package will be unlisted and we will bring back the single NuGet when the reported gaps are addressed.

[creyke]

While I don't share @jackbond's thirst for blood just yet, it would be great to know when this will be fixed. It's blocking a major piece of work for me so I'd like to know at least if it's going to be hours, days, or weeks? Thanks.

[ausfeldt]

This thread has much more context than original issue started with. If you are looking for fixes in 1.21.0, today we will publish a new version with-out (single-nuget changes). @creyke will this unblock you?

[creyke]

Thanks @ausfeldt - will confirm when I’ve tested it.

[creyke]

@ausfeldt The version just pushed (1.21.1) doesn't compile for me when referenced by a netstandard2.0 project...

[ausfeldt]

@creyke 1.21.1 does not target .net standard anymore. Those changes were rolled back. https://www.nuget.org/packages/Microsoft.Azure.DocumentDB.Core is the package that supports .net standard.

[creyke]

I get this same error message in the latest sdk (tools: 2.0.1-beta.23, runtime: 2.0.11587.0)...

[12/03/2018 17:20:59] A ScriptHost error has occurred
[12/03/2018 17:20:59] The listener for function 'Function1' was unable to start. System.Private.CoreLib: One or more errors occurred. (Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)). Microsoft.Azure.DocumentDB.Core: Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E).

...with only the following references...

  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.CosmosDb" Version="3.0.0-beta6" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.9" />
  </ItemGroup>

...as well as a CosmosDBTrigger reference...

        [FunctionName(nameof(Function1))]
        public static void Run([CosmosDBTrigger(
            databaseName: "databaseName",
            collectionName: "collectionName",
            ConnectionStringSetting = "connectionString",
            LeaseCollectionName = "leases")]IReadOnlyList<Document> input, TraceWriter log)
        {
            if (input != null && input.Count > 0)
            {
                log.Verbose("Documents modified " + input.Count);
                log.Verbose("First document Id " + input[0].Id);
            }
        }

[ausfeldt]

@creyke The fix was in the single nuget which was rolled back. Here is a .net standard package which has the fix. Microsoft.Azure.DocumentDB.Core.1.9.1-dev-private.nupkg.

[creyke]

@ausfeldt Thanks for uploading the dev package, however when I try and install it I get the following error, and it also does not seem to be a valid zip file...

[Local] 'C:\Working\NuGetMicrosoft.Azure.DocumentDB.Core.1.9.1-dev-private.nupkg' is not a valid nupkg file. Error: 'End of Central Directory record could not be found.'.

[ausfeldt]

@creyke I just tried and was not able to reproduce. https://github.com/Azure/azure-documentdb-dotnet/raw/DevTestPackages/DevTestPackages/Microsoft.Azure.DocumentDB.Core.1.9.1-dev-private.nupkg

Install-Package

I would try one more time. Maybe the download was corrupt.

[creyke]

@ausfeldt Thanks again, I'm sorry for the confusion, it was corrupt. After referencing it...

    <PackageReference Include="Microsoft.Azure.DocumentDB.Core" Version="1.9.1-dev-private" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.CosmosDb" Version="3.0.0-beta6" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.9" />
    <PackageReference Include="System.Data.SqlClient" Version="4.4.0" />

...I still unfortunately get the same error...

[12/03/2018 21:46:57] A ScriptHost error has occurred
[12/03/2018 21:46:57] The listener for function 'Function1' was unable to start. System.Private.CoreLib: One or more errors occurred. (Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)). Microsoft.Azure.DocumentDB.Core: Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E).

[creyke]

Sorry to chase, any options here? I am unable to deploy my code to Azure Functions so am completely blocked.

[ausfeldt]

@creyke Is the azure function 64-bit running on windows?

[isaacabraham]

I'm seeing the same issue in with F# scripts against 32-bit FSI. I can't load Microsoft.Azure.Documents.ServiceInterop.dll since it is 64-bit only. Is the DocDb client nuget package 64-bit only?

Interestingly I only see this issue if I try to do an IQueryable query. Simple getting all rows works:

let myQuery = client.CreateDocumentQuery<Row>(pricesCollection.SelfLink)

myQuery |> Seq.toArray // works

query {
    for doc in myQuery do
    take 10
    select (doc.Price, doc.Town) }
|> Seq.toArray // pop

Any ideas?

[ausfeldt]

@isaacabraham The nuget package is not 64-bit only. Microsoft.Azure.Documents.ServiceInterop.dll is 64-bit and native. The SDK will only use it if it's running on a Windows 64-bit OS.

[jackbond]

@ausfeldt I'm curious, given the agony that ServiceInterop.dll has created, is the performance boost on 64 bit windows worth it? Also, if you can run without it, can you guys simply catch the Exception and fall back gracefully?

[ausfeldt]

@jackbond This option is an option that is being considered. Long term we are planing to get rid of this dependency all together. Thanks.

[creyke]

@ausfeldt Apologies for no response to previous question. I ran out of time and had to remove my dependencies on cosmos unfortunately. Will monitor the thread, but I no longer have a reproduceable case as I had to ship something (have gone back to http for now instead of cosmos db triggers).

[isaacabraham]

@ausfeldt I'm running on 64 bit Windows 10. I can't reference those two interop libraries - it says Bad CLI Header. Is there no way to "disable" this 64 bit optimisation?

[ausfeldt]

@isaacabraham There is currently no way to disable. The Nuget Package adds build props to make sure that the native DLL is copied to your bin directory. Native assemblies can't be referenced using the #r and #I. The working directory of the F# Interactive process is the path where it is installed, which is somewhere in C:\Program Files, where it will not find the DLL. You options are:

Set the working directory where the DLL is:

System.Environment.CurrentDirectory <- __SOURCE_DIRECTORY__

Windows also considers the Path environment variables to look for the DLL:

Environment.SetEnvironmentVariable("Path", Environment.GetEnvironmentVariable("Path") + ";" + __SOURCE_DIRECTORY__)

[isaacabraham]

@ausfeldt thanks for this. I'd say there are a few issues here:

1. Build props obviously are only of use with a console application, rather than a script.
2. Whilst the NuGet client may add build props, I don't believe Paket does.

The actual working directory for FSI isn't in C:\Program Files etc. - it varies depending on which IDE you use - but in VS Code, it's already set to the __SOURCE_DIRECTORY__ macro, and it isn't able to find the dll. However, I was able to get it working by adding the dll folder to the path e.g.

let x64 = IO.Path.GetFullPath @"packages\cosmos\Microsoft.Azure.DocumentDB\runtimes\win7-x64\native\"
Environment.SetEnvironmentVariable("Path", Environment.GetEnvironmentVariable "Path" + ";" + x64)

Thanks for your help here.

[spelltwister]

@ausfeldt I'm a little confused by these two comments:

Microsoft.Azure.DocumentDB.Core has been deprecated. Can you try 1.21.0 https://www.nuget.org/packages/Microsoft.Azure.DocumentDB/

@creyke 1.21.1 does not target .net standard anymore. Those changes were rolled back. https://www.nuget.org/packages/Microsoft.Azure.DocumentDB.Core is the package that supports .net standard.

Is the .Core package deprecated? Seems like they were both updated few weeks ago. Just want to make sure I'm not using a package that is now dead.

I have read thru the entire chain and tried all the suggestions I could apply to an asp.net core 2.0 site. Every package version fails with the same error "Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)". I tried to also include the fx version which seems to have helped someone else, but no luck on my end. I also tried updating permissions on the directory and gave Everyone Full Control, but no luck. Works fine when run via unit test, just doesn't work at all behind a webapi.

My only option at this point is to change my .netstandard lib to fx and reference the other DocumentDB package; adjusting any differences in code. Not sure if I can host this in an asp.net core 2.0 site or if needs to be asp.net fx. Hopefully that will at least let me ship this project. Kinda sucks that all the development was done against the unit test project until tests were passing only to find out the host has problems with this package T__T

[ausfeldt]

@spelltwister Can you give this package a try. https://github.com/Azure/azure-documentdb-dotnet/raw/DevTestPackages/DevTestPackages/Microsoft.Azure.DocumentDB.2.0.0-preview.nupkg

[spelltwister]

Unfortunately, no luck with the new package. I tried updating just the asp.net core 2.0 references assuming it would use the newer version, but same error. I also tried updating both the .netstandard 1.6 service library and the asp.net core 2.0 site references, but same error. :( Will be trying to convert back to full framework lib and full framework asp.net site.

[spelltwister]

I was able to convert to using the fx package with no code changes required; just copied all the files into new full framework project so that's cool. The unit tests all still passed after making that change. I was able to do a basic test to see that the service does work when hosted in a asp.net full framework webapi project. However, I'm having other problems now due to System.Runtime not being found, but that's a different issue.

[ausfeldt]

@jackbond Here are the packages that will gracefully fallback if the native assemblies are not found.
Microsoft.Azure.DocumentDB.1.21.1.nupkg and Microsoft.Azure.DocumentDB.Core.1.9.1.nupkg

[jackbond]

@ausfeldt Thanks Garrett. I managed to resolve this issue a while back by simply adding the DocDb nuget to all my executable projects, but still receive notifications on this issue. I'll be happy when the native stuff is gone entirely, as they seem to make little sense for this sort of project.

[r3plica]

I have started getting this issue now.. I had a working project on my master branch (which was published about a month ago) and today I needed to update my web.config (unrelated) and I started getting this error:

"Unable to find an entry point named 'CreateServiceProvider' in DLL 'Microsoft.Azure.Documents.ServiceInterop.dll'."

I have tried everything in this post and everything in the referenced post and I am still getting this error.
For reference, I have 3 projects with the Microsoft.Azure.DocumentDB package installed (v2.0.0): Project.DocumentDb (which has the service and repository), Project.Web which references Project.DocumentDb, so I added the package as suggested above and finally Project.Api which is the API with controllers, etc. This references Project.Web, so again I added the package as stated above.

I have checked the bin folder and both DLLs are present. This is annoying, because it just started happening .....

[ealsur]

@r3plica Did you restart the App? That error can also appear if the ServiceInterop.dll is locked by some process.

[mcupito]

@ealsur What could be locking that .dll? Can you provide some examples or things to look for in code?

[ealsur]

@mcupito Lock in DLLs in Azure App Service sometimes occur if there is an ongoing process using it, sometimes due to an underlying runtime upgrade (sometimes App Service sends updates to the framework) and I've seen it also happen on deployments. I'm not 100% sure of the reason though, just that an App Service restart unlocks it.

[Adam4224]

I was able to fix this error when running locally by adding the following to my web.config under the system.web section:

<hostingEnvironment shadowCopyBinAssemblies="false" />

Absolutely FANTASTIC. That was 2 days of absolute hell trying to get the new DocumentDB libraries working on the Azure app service. Fixed just like that!!! Fantastic!

[kirankumarkolli]

Thanks @Adam4224 .

@j82w lets please capture it in troubleshooting FAQs.