Confluent-kafka-dotnet: System.DllNotFoundException: Unable to load DLL 'librdkafka' (Exception from HRESULT: 0x8007007E)

[manpreet845]

System.DllNotFoundException: Unable to load DLL 'librdkafka' (Exception from HRESULT: 0x8007007E)

I just wanted to note that running an application with this library, requires Visual C++ Redistributable Packages for Visual Studio 2013.
I couldn't find any Documentation for this requirement, just letting you know since the exception is misleading.

[mhowlett]

thanks a lot for pointing that out. I thought we were statically linking these - @edenhill ?

[edenhill]

@manpreet845 which Confluent.Kafka and librdkafka.redist nuget package versions are you on?

[manpreet845]

@edenhill i was on 0.9.2-preview2 updated to 0.9.4-preview4 and had the same problem.

package id="Confluent.Kafka" version="0.9.4-preview4" targetFramework="net461"
package id="librdkafka.redist" version="0.9.4-RC1-preview4" targetFramework="net461"
package id="RdKafka.Internal.librdkafka" version="0.9.2-ci-28" targetFramework="net461"

[mhowlett]

please try referencing nuget package Confluent.Kafka 0.9.4-preview8. we think we've probably solved this, but haven't verified.

[manpreet845]

@mhowlett I tried Confluent.Kafka 0.9.4-preview8, but you still need the c++ runtime environment installed

[mhowlett]

So you had the problem on 0.9.4-preview2 as well? that means the RdKafka.Internal.librdkafka has the same requirement (that package wasn't created by us).

Are you on 64 or 32 bit? What OS specifically?

we will try to replicate.

[manpreet845]

Sorry for the confusion I did mean the depency RdKafka.Internal.librdkafka, we have machines that are provisioned with puppet and we only install what we need. Our stack is mainly c# so this error wasn't obvious to debug. I have since installed the c++ run time on those machines, so its not an issue for us anymore.
Just wanted to let you know incase some one else ran into the same issue

[mhowlett]

hi @manpreet845 - feedback very much appreciated. We want Confluent.Kafka to just work out of the box and don't want to have the Visual C++ Redistributable Packages for Visual Studio 2013 dependency (at this point we're not quite sure where that's coming in). This issue is pretty high priority for us, we'll look into it again soon.

[jhuggart]

I'm seeing the same exception on CentOS 7 when instantiating a producer.

I'm using version 0.9.4 of both Confluent.Kafka and librdkafka.redist. Do you guys have any suggestions on how to resolve?

[edenhill]

@jhuggart It could be that librdkafka's dependencies are missing, primarily openssl and libsasl.
Can you provide the output of ldd ~/.nuget/packages/librdkafka.redist/0.9.4/runtimes/rhel-x64/native/librdkafka.so ?

[jhuggart]

It looks like I need liblz4 and libsasl. Here's the output:

gcc librdkafka.so
/usr/bin/ld: warning: liblz4.so.1, needed by librdkafka.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libsasl2.so.2, needed by librdkafka.so, not found (try using -rpath or -rpath-link)
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
librdkafka.so: undefined reference to `LZ4F_compressUpdate'
librdkafka.so: undefined reference to `sasl_listmech'
librdkafka.so: undefined reference to `sasl_errstring'
librdkafka.so: undefined reference to `LZ4F_compressBegin'
librdkafka.so: undefined reference to `sasl_getprop'
librdkafka.so: undefined reference to `sasl_client_init'
librdkafka.so: undefined reference to `LZ4F_createDecompressionContext'
librdkafka.so: undefined reference to `LZ4F_freeDecompressionContext'
librdkafka.so: undefined reference to `sasl_client_start'
librdkafka.so: undefined reference to `LZ4F_compressBound'
librdkafka.so: undefined reference to `sasl_dispose'
librdkafka.so: undefined reference to `LZ4F_isError'
librdkafka.so: undefined reference to `LZ4F_decompress'
librdkafka.so: undefined reference to `LZ4F_getErrorName'
librdkafka.so: undefined reference to `LZ4F_compressEnd'
librdkafka.so: undefined reference to `sasl_errdetail'
librdkafka.so: undefined reference to `sasl_client_step'
librdkafka.so: undefined reference to `LZ4F_getFrameInfo'
librdkafka.so: undefined reference to `LZ4F_createCompressionContext'
librdkafka.so: undefined reference to `LZ4F_freeCompressionContext'
librdkafka.so: undefined reference to `sasl_client_new'
collect2: error: ld returned 1 exit status

[edenhill]

Aye, right! install cyrus-sasl and lz4

[jhuggart]

That did it!

For anyone that stumbles upon this, I also needed to symlink libsasl2.so.3 --> libsasl2.so.2 to make librdkafka happy.

[mhowlett]

Can we provide these in the NuGet package? Statically link?

At the very least I think we need to throw an informative exception (which will involve explicit checks for dependent libraries).

[edenhill]

@jhuggart Ooh, linking across SONAME versions (3 to 2) is not safe. The version bump indicates that there is no ABI compatibility and you might see weird behaviour, crashes, etc.
If you are not actually using SASL authentication then this is not an issue, but if you are you will probably need to install an older version of cyrus-sasl or install a librdkafka version that was compiled for CentOS 7 (or build one yourself, easy enough). Let me know if you need assistance with that.

[mhowlett]

or could we package them in a separate nuget package on which librdkafka.redist depends?

[edenhill]

@mhowlett

Static linking

Static linking of OpenSSL is generally considered a bad idea. For SASL it is not really feasible since the actual authentication modules are loaded during runtime. LZ4 should probably be linked statically though.

Bundling

Bundling dynamic libraries, such as we do on Windows, is hard since each of those libraries also have dependencies, and their dependencies have dependencies. And some of those are bound to be incompatible with the host OS at some point (just look at this issue where librdkafka was built for CentOS 6 which has all the same packages as CentOS 7, but obviosly with some version mismatches - which is effectively the same thing as not available).

Weak linking

There's been some discussions on weak-linking librdkafka dependencies so that the code can detect if a library is available and then nicely/noisely disable features instead of just not starting.
It solves the case where people don't need the functionality provided by those libraries, and at least provides some indication of what is missing, through a nice error message, when the user tries to use the disabled functionality.

All of this was solved in RPM and DEB about 20 years ago but new hip packaging systems (yes, looking at you NuGet, and Go..) blindly ignores this fact and assumes it isn't a problem that needs to be solved.

[mhowlett]

right. for the next release I vote for trying to load the dependencies individually before trying to load librdkafka and throwing appropriate exceptions if they fail (this should be quick to implement).

I like the weak linking idea. you could do it at a higher level too by providing a version of librdkafka with openssl/sasl disabled and a version with it enabled, and loading the former if there are problems loading the latter.

[edenhill]

loading the deps individually requires dlopen()/LoadLibrary() and it will be fragile.

The weak linking is more robust and much easier to implement (just check func pointer for zero).

[jhuggart]

@edenhill Thanks for the info. I'm not using SASL right now. But I don't like the idea of an unsafe link.

Is it possible to update librdkafka.runtimes with the library for centos7 added in runtimes/native/centos.7-x64?

[edenhill]

@jhuggart You mean that the librdkafka.redist package should provide more specific runtimes where SONAMEs actually match?

[edenhill]

Also, the linking in itself is not a problem (unless the linking fails, which it didnt).
It is when linked functions are being called that the SONAME incompatibility kicks in and ruins your day, but apart from one single global init call there are no functions being called unless you configure the client to use SASL - so you should be safe.

[jhuggart]

@edenhill That's what I was thinking. Unless I'm missing something, there is not a cyrus-sasl package that includes libsasl.so.2 built for centos7. So including the specific runtime shouldn't cause any harm.

FWIW, I'm no expert in packaging runtimes for cross-platform distribution. So if I'm totally off base, just let me know.

[edenhill]

@jhuggart I'll create an issue on librdkafka to make sure the next nuget package includes a librdkafka library built for CentOS 7 as well.

[jbezak]

I ran into this issue today while trying the fsharp example with latest nuget package
I am using Any CPU and "Prefer 32 bit" set to true and I have noticed that there is x86 and x64 folder within the Debug/Release one. When I manually copy the x86 folder content directly into Debug/Release the exception is gone. I am not sure why the process is not looking into the x86 folder by itself.

[treziac]

@jbezak what framework did you target, and what is your ide / nuget version?
I assume it's on windows
x86/x64 floder are for the net45 version of the lib only, not the netstandard1.3 one. Don't know fsharp enough to know which one it use

[vaibhavg1985apr]

Hello,

I am getting same error for windows 7. Could you please let me know what i have to install to make this working.

Unable to load DLL 'librdkafka': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

[treziac]

What is your target framework & platform?
On windows, you might need Microsoft Visual C++ 2013 Redistributable x86 / x64 if it's no installed

[vaibhavg1985apr]

Framework: 4.5.1
Windows 7
Already have Microsoft Visual C++ 2013 Redistributable x86 / x64

[treziac]

What package version do you use?
Does it happen in your IDE (which one) or when launching an app?
Does the exception occur as soon as you call the lib or only on some specific call? (Flush...)
Do you have a x86/x64 folder with dll next to your executable?

If it's still not working, could you check in your code that

            var baseUri = new Uri(Assembly.GetExecutingAssembly().GetName().CodeBase);
            var baseDirectory = Path.GetDirectoryName(baseUri.LocalPath);
            var directory = Path.Combine(baseDirectory, "x64")

directory actually point to the x64 folder?

[vaibhavg1985apr]

1. package version: 0.9.4
2. IDE: VS2013.
3. Please see attachment for exact line of code where i am getting this exception and exact exception message.
   4: Yes. I see x64 directory.

By the way, i already have OpenSSL installed on my system.

kafka Error.txt

[treziac]

It's not unable to load library, but a problem with your config

You have auto.offset.reset twice, remove the one on the root of the config (it's a topic configuration)

[vaibhavg1985apr]

Got it. I tried and it worked. Thanks you for your help!

[afrancoc2000]

Hi, I'm having the same problem in version v0.11.2 in an azure function. I get this error:

[11/28/2017 1:15:32 PM] C# Timer trigger function executed at: 11/28/2017 8:15:32 AM
[11/28/2017 1:15:35 PM] A ScriptHost error has occurred
[11/28/2017 1:15:35 PM] Exception while executing function: SaveAliasChanges. Confluent.Kafka: The type initializer for 'Confluent.Kafka.Impl.LibRdKafka' threw an exception. Confluent.Kafka: Unable to load DLL 'librdkafka': The specified module could not be found. (Exception from HRESULT: 0x8007007E).
[11/28/2017 1:15:35 PM] Exception while executing function: SaveAliasChanges
[11/28/2017 1:15:35 PM] Exception while executing function: SaveAliasChanges. Confluent.Kafka: The type initializer for 'Confluent.Kafka.Impl.LibRdKafka' threw an exception. Confluent.Kafka: Unable to load DLL 'librdkafka': The specified module could not be found. (Exception from HRESULT: 0x8007007E).
[11/28/2017 1:15:35 PM] Function completed (Failure, Id=efc2ac4a-8390-4ddd-ad69-332da64bf190, Duration=5536ms)
[11/28/2017 1:15:35 PM]
[11/28/2017 1:15:35 PM] Executed 'SaveAliasChanges' (Failed, Id=efc2ac4a-8390-4ddd-ad69-332da64bf190)
[11/28/2017 1:15:35 PM] mscorlib: Exception while executing function: SaveAliasChanges. Confluent.Kafka: The type initializer for 'Confluent.Kafka.Impl.LibRdKafka' threw an exception. Confluent.Kafka: Unable to load DLL 'librdkafka': The specified module could not be found. (Exception from HRESULT: 0x8007007E).
[11/28/2017 1:15:35 PM]   Function had errors. See Azure WebJobs SDK dashboard for details. Instance ID is 'efc2ac4a-8390-4ddd-ad69-332da64bf190'
[11/28/2017 1:15:35 PM] mscorlib: Exception while executing function: SaveAliasChanges. Confluent.Kafka: The type initializer for 'Confluent.Kafka.Impl.LibRdKafka' threw an exception. Confluent.Kafka: Unable to load DLL 'librdkafka': The specified module could not be found. (Exception from HRESULT: 0x8007007E).

any ideas?

[mhowlett]

@afrancoc2000 - I've not deployed an azure function before, but I'm guessing you're targeting .NET Framework and the x86/x64 directories containing librdkafka.dll are not being copied over to the same directory as Confluent.Kafka.dll. I'll add a task for myself to try this deployment scenario to investigate.

Also, you can try the CI build 0.11.2-ci-164 which includes changes to library loading (and may work out of the box). It also allows you to explicitly specify a path to librdkafka using Library.Load. This will be in an official release very soon, possibly tomorrow.

[afrancoc2000]

Thats great thank you @mhowlett, I'm trying to download the CI build 0.11.2-ci-164 but I haven't been able to find it in the prerelease nugets, do I have to compile it myself?

[afrancoc2000]

Copying the files manually to the bin folder worked. Thank you very much :)

[mhowlett]

Closing as the original issue should be resolved. There are still scenarios where System.DllNotFoundException can happen, but 0.11.3 includes Library.Load as a workaround and all cases I'm aware of that we intend to fix have been fixed.

[afrancoc2000]

Tried the new nuget v0.11.3 it works great! thanks!