Confluent-kafka-dotnet: System.InvalidOperationException: Error while loading librdkafka.dll or its dependencies

[edudelta]

Description

I'm facing this issue when trying to run a test case.

How to reproduce

Environment:

• Windows 10 x64
• Visual Studio Enterprise 2017 (both 15.5.3 and 15.5.4)
• .NET 4.6.1
• Confluent.Kafka 0.11.3 livrdkafka.redist 0.11.3

Message: Test method Integration.Tests.ProjectEventHandlerTests.FirstTest threw exception:
System.InvalidOperationException: Error while loading librdkafka.dll or its dependencies from C:Usersedusourcerepos...Integration.TestsTestResultsDeploy_edu 2018-01-17 16_11_05Outruntimeswin7-x86nativelibrdkafka.dll. Check the directory exists, if not check your deployment process. You can also load the library and its dependencies by yourself before any call to Confluent.Kafka ---> System.ComponentModel.Win32Exception: The specified module could not be found

[mhowlett]

Thanks - I've run into this as well doing integration tests for Avro - librdkafka is not automatically deployed for .NET Framework test projects (though everything is fine using .NET Core). At this stage I'm not sure if this is easily fixed or whether it's a more fundamental limitation with how nuget works in the .NET Framework case.

[grhm]

I'm having the same issue - and am using a slight rewrite of the AdvancedConsumer example.
I'm in a phase where I'm starting many short lived debug sessions, and may not always be cleanly closing the session.
I've found if I edit the code to change my group.id to something else it works again for a while. When it then errors again, I can name it back and all is good for a while.

In the logs just before it errors, I'm seeing things like:

Group MyTestReaderGroup1 with generation 2 is now empty

Not sure if this helps pin down the issue (either with the lib, or with my code).

[mhowlett]

@grhm - what if you make some other change to the code, does that fix it as well? seems like a different scenario where librdkafka.dll isn't getting deployed properly by VS for some reason (but this time an intermittent problem).

[grhm]

Ok. Had a change to check this now. When I have the error, if I do a dir /s/b *kafka* I get:

C:CodeMyProjectbinDebugConfluent.Kafka.dll
C:CodeMyProjectbinDebugConfluent.Kafka.xml
C:CodeMyProjectbinDebuglibrdkafka

i.e. an empty librdkafka directory.
I've then changed the code, and rebuilt via Visual Studio's F5 command to start debugging and the code works. Doing another dir gives me:

C:CodeMyProjectbinDebugConfluent.Kafka.dll
C:CodeMyProjectbinDebugConfluent.Kafka.xml
C:CodeMyProjectbinDebuglibrdkafka
C:CodeMyProjectbinDebuglibrdkafkax64librdkafka.dll
C:CodeMyProjectbinDebuglibrdkafkax64librdkafkacpp.dll
C:CodeMyProjectbinDebuglibrdkafkax86librdkafka.dll
C:CodeMyProjectbinDebuglibrdkafkax86librdkafkacpp.dll

i.e. librdkafka dll's in the previously empty directory.
@mhowlett I tried this with a code change that was white space only - and that fixed it too - so as you suggested it's librdkafka.dll isn't getting deployed properly, but I'm none the wiser as to why.

[QianJiali]

I have the same problem.
I tried a lot then finally fixed with adding this to .csproj
<Import Project="$(USERPROFILE)\.nuget\packages\librdkafka.redist\0.11.3\build\librdkafka.redist.props" Condition="Exists('$(USERPROFILE)\.nuget\packages\librdkafka.redist\0.11.3\build\librdkafka.redist.props')" />

It works fine now.

[edudelta]

Hi!
Just to inform, the solution purposed by QianJiali was not enough to solve the issue, by my side.
Thanks

[teixeirajmp]

I also encountered this issue and have no proper solution yet. As I'm sure .NET Framework will be with us for some time, it would be nice to have a way to solve or work around this issue.

[teixeirajmp]

I found a workaround for my particular circustances.
In our case, the issue was happening when running continuous integration and the CI task uses MSTest.exe as test runner.
The problem is that MSTest doesn't recognize that the native libraries are something it needs to copy to its work directory.
The most straightforward solution would be to create a .testsettings file with a Deployment section, like so:

<TestSettings name="" id="" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <!-- etc. -->
  <Deployment>
    <DeploymentItem filename="packages\librdkafka.redist.0.11.3\runtimes\win7-x86\native\msvcr120.dll" outputDirectory="librdkafka\x86" />
    <DeploymentItem filename="packages\librdkafka.redist.0.11.3\runtimes\win7-x86\native\librdkafka.dll" outputDirectory="librdkafka\x86" />
    <DeploymentItem filename="packages\librdkafka.redist.0.11.3\runtimes\win7-x86\native\zlib.dll" outputDirectory="librdkafka\x86" />
    <DeploymentItem filename="packages\librdkafka.redist.0.11.3\runtimes\win7-x86\native\librdkafkacpp.dll" outputDirectory="librdkafka\x86" />
  </Deployment
  <!-- etc. -->
</TestSettings>

and modify the CI task adding the switch /testsettings:

mstest  /testcontainer:Tests.dll ...  /testsettings:Tests.testsettings

If you can't use a .testsettings (e.g. you might not have access to the CI task configuration), another work around is to decorate all the test classes that use kafka with DeploymentItemAttribute.

[wilfriedb]

I also run into his issue for an ASP.NET standard (not Core) website. The file is apparently expected at:
websitebinruntimeswin7-x64nativelibrdkafka.dll
but it is actually at:
websitebinlibrdkafka.dll

This is something I have also seen with another package. Maybe it is a regression in the ASP.NET framework, but it would be nice if there is an easy solution for this.

[mohoch1]

We are having the same issue with a .NET application.
The file is expected to be in runtimeswin7-x64nativelibrdkafka.dll but is actually in bin/librdkafka
It was working fine with the earlier versions of confluent-kafka-dotnet.

Is there any way of fixing/configuring it?

[felipecalvo]

I'm having the same issue here. What I've found solves it is manually copying the runtimes folder found at

C:Users\{USERNAME}\.nugetpackageslibrdkafka.redist\{VERSIONNUMBER}

to the

{PATHTOPROJECT}bin{ENVIRONMENT}

folder.

It's probably an issue with PackageReference.

[mhowlett]

thanks for reporting everyone. noted, will investigate.

[xs2bharat]

I ran into same issue and i was using 0.11.3 nuget package as well. In my case i am using Confluent.Kafka.StrongName and it was expecting librdkafka.dll at C:WindowsMicrosoft.NETassemblyGAC_MSILConfluent.Kafka.StrongNamev4.0_0.11.3.0__12c514ca49093d1eruntimeswin7-x64native. After copying it manully, it worked but earlier copied them in system32 and syswow64 folder which used to be good enough.

[ms0713]

@mhowlett
Have you found any solution?
If I use librdkafka.dll, do I need any kind of licensing while I go for production?

[mhowlett]

I haven't looked further yet.

librdkafka and the confluent .net client are both apache licensed. you don't need any additional licenses for production use.

[ms0713]

@mhowlett
librdkafka and confluent-kafka both shows different License Info.
librdkafka license info does not mention any Apache thing.

I think @edenhill can help for this.

[mhowlett]

https://github.com/edenhill/librdkafka/blob/master/LICENSES.txt
https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/LICENSE

[adison88]

Is there any solution or work around on this? I'm also facing this while using .net client in Framework 4.6.2 whereas it works fine on .Net Core.

[mhowlett]

it might be worth trying with the 1.0-experimental-8 nuget package as RIDs have been changed from win7-x64/x86 to win-x64/x86, and this may be the reason the binaries weren't being copied over. not sure though, haven't tried.

[mhowlett]

or changes may be required here, if in fact the library is getting copied but to an unexpected location: https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/src/Confluent.Kafka/Impl/LibRdKafka.cs#L221

[adison88]

created a PR for the scenario when DLLs are getting copied inside directory path.
https://github.com/confluentinc/confluent-kafka-dotnet/pull/589

[mhowlett]

589 has been merged into 1.0-experimental, and will be incorporated in nuget packages >= 1.0-experimenal-8.

[SaibabaGajulawar]

Is this resolved ?

[RasicN]

I'm still getting this error on the v1.0.0-beta3

System.InvalidOperationException
  HResult=0x80131509
  Message=Error while loading librdkafka.dll or its dependencies from C:\Projects\Kafka\ConsoleApp1\bin\Debug\librdkafka.dll. Check the directory exists, if not check your deployment process. You can also load the library and its dependencies by yourself before any call to Confluent.Kafka
  Source=Confluent.Kafka
  StackTrace:
   at Confluent.Kafka.Impl.Librdkafka.Initialize(String userSpecifiedPath)
   at Confluent.Kafka.Consumer`2..ctor(ConsumerBuilder`2 builder)
   at Confluent.Kafka.ConsumerBuilder`2.Build()
   at ConsoleApp1.Program.Main(String[] args) in C:\Projects\Kafka\ConsoleApp1\Program.cs:line 28

Inner Exception 1:
Win32Exception: %1 is not a valid Win32 application

It occurs at runtime when doing the following:
var c = new ConsumerBuilder<string,string>(conf).Build();

Tried the above solutions to no avail.

[gzinger]

I am on version 1.4.2 and still having the same issue. Perhaps the dependent library could be included in the main .nuget package?

[Seamless2014]

I Having the same issue.

System.InvalidOperationException: Error while loading librdkafka.dll or its dependencies from D:DebugCodeNETServerbinDebuglibrdkafka.dll. Check the directory exists, if not check your deployment process. You can also load the library and its dependencies by yourself before any call to Confluent.Kafka ---> System.ComponentModel.Win32Exception: 找不到指定的模块。

I use .net Framework 4.5

Anybody Resolve the issue?

Thanks