Confluent-kafka-dotnet: System.DllNotFoundException: Failed to load the librdkafka native library

[merlin-king]

System.DllNotFoundException: Failed to load the librdkafka native library

An error occurred using the Dotnet core Web API
Errors are as follows

Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HL9RN5T1D654", Request id "0HL9RN5T1D654:00000001": An unhandled exception was thrown by the application.
System.DllNotFoundException: Failed to load the librdkafka native library.
at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath)
at Confluent.Kafka.Producer..ctor(IEnumerable1 config, Boolean manualPoll, Boolean disableDeliveryReports) at Confluent.Kafka.Producer2..ctor(IEnumerable1 config, ISerializer1 keySerializer, ISerializer`1 valueSerializer, Boolean manualPoll, Boolean disableDeliveryReports)

How to reproduce

Use Asp.net Core WebApi on Centos7

[mhowlett]

@merlin-king - we do not currently include a build of librdkafka for centos, though in the future we plan to support this platform. you can either run your application within docker (microsoft's images are debian based), or specify the location of librdkafka manually using the Library.Load method (available in version 0.11.3). You can get a suitable build for centos using Confluent's packages, or build it yourself https://github.com/edenhill/librdkafka#building

[TheMidgardWatcher]

Hi there!

I have similar issue on Openshift+RedHat docker images.
After I used Library.Load method I got such error message:

InvalidOperationException: Failed to load librdkafka at location '/opt/app-root/app/librdkafka/x64/librdkafka.dll'. dlerror: '/opt/app-root/app/librdkafka/x64/librdkafka.dll: invalid ELF header'.
Confluent.Kafka.Impl.LibRdKafka.Initialize(string userSpecifiedPath)

I assume that the dll was found but not loaded correctly.

Also i can add that we use OpenShift's s2i-dotnetcore images with RedHat OS and this builder runs
dotnet publish command and I'm not sure if it publishes it correctly to get it work.

I'll appreciate any help on this "dll loading" issue!

[mhowlett]

in the future we plan to support RedHat transparently, but for now you'll need to use Library.Load.

the shared library for RedHat should have an extension .so, i.e. librdkafka.so. Where are you getting the binary from? you can use the confluent packages https://docs.confluent.io/current/installation/clients.html#c-c or install from source https://github.com/edenhill/librdkafka#building

[mhowlett]

if the library is installed via our packages or from source as instructed it will be in a standard path, and I think you should be able to load it using just Library.Load("librdkafka.so") (note: if you just installed it, you may need to call ldconfig to update the cache).

or you can deploy a redhat compatible binary along with your app and use an absolute / relative path.

[TheMidgardWatcher]

@mhowlett I'm not sure if i understood you correctly. Let me clarify what i mean:

• I have some repo with aspnet core application
• And I have configured OpenShift application (wich has Rhel images as it's base)
• In that application I have github hook configured to listen pushes to my repo, if it receives push, it starts building and deploying application

I'm not building or configuring anything in base image. But the problem is that dotnetcore image builder (OpenShift's s2i-dotnetcore) publishes application without concrete platform, so there are no 'so' libs at all out of the box.

As I understood you - in my case I need 'so' library in some place in my solution folder. I see 2 variants:
First - NuGet package should contain 'so' files in specific platform folder, like for instance 'win-x64' contains dll's.
Second way - I need to publish my application for target platform and then copy and include in project 'so' files (and do this after each update of package)

In both cases I need to load this lib manually in code if it running inside OpenShift environment.
Can you confirm if I understand you correctly?

PS: I'll try to go second way later today.

[mhowlett]

@TheMidgardWatcher correct. You'll need to get a librdkafka.so that is compatible with the docker image. I suggest compiling from source inside a docker container of the type you will be deploying to (you'll need to install g++ and some other packages to do this probably). Then include this librdkafka.so in your project and make sure it's deployed with your project. then reference this as a parameter to Library.Load.

[TheMidgardWatcher]

Hey @mhowlett ,

Just tried to publish project on my win workstation to get this 'so' libraries. I got them and put to root directory of the project. It was published to OpenShift successfully, but when I tried to call Library.Load
I got error message again:

**An unhandled exception occurred while processing the request.**
InvalidOperationException: Failed to load librdkafka at location '/opt/app-root/app/librdkafka.so'. dlerror: ''.
Confluent.Kafka.Impl.LibRdKafka.Initialize(string userSpecifiedPath)

I've checked files inside of deployed POD - they are there:

D: >oc rsh dev-netcore-36-hvk3k ls -lhS /opt/app-root/app/
total 41M
-rw-rw-rw-  1 default root 5.8M Dec  6 10:37 librdkafka.so
-rw-rw-rw-  1 default root 5.2M Dec  6 10:37 debian9-librdkafka.so
[....]
-rw-rw-rw-  1 default root 858K Dec  6 10:37 libe_sqlite3.so
[...]
-rw-rw-rw-  1 default root 463K Dec  6 10:37 libuv.so

Before start using confluent.kafka (e.g. connection to brokers and producers/consumers creation)
Right in WebAPI controller I'm doing this:

            var isLoaded = Library.IsLoaded;
            if (!isLoaded && !_env.IsEnvironment("Debug"))
            {
                var pathToLibrd = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location),
                    "librdkafka.so");
                Console.WriteLine($"Trying to load {pathToLibrd}");
                Library.Load(pathToLibrd);
                Console.WriteLine($"Using librdkafka version: {Library.Version}");
            }

What am I doing wrong this time? And what steps I need to take to make it work?

[TheMidgardWatcher]

Addition:
Now I'm facing this error message:

An unhandled exception occurred while processing the request.

InvalidOperationException: Failed to load librdkafka at location '/opt/app-root/app/librdkafka.so'. dlerror: 'libcrypto.so.1.0.0: cannot open shared object file: No such file or directory'.
Confluent.Kafka.Impl.LibRdKafka.Initialize(string userSpecifiedPath)

[edenhill]

Try installing OpenSSL 1.0.x

[TheMidgardWatcher]

Hi @edenhill ,

Since I'm using openshift and RedHat's s2i-dotnetcore builder image to run my aspnet core 2.0 application it wouldn't be so easy... But if there are no other options I'll try to add openssl lib to application template tomorrow, because I don't see any possibility to do it another way. I can't do anything inside of running container...

[mhowlett]

if you don't need ssl or sasl, you could compile a version of librdkafka that doesn't include these features that (i believe) won't depend on libcrypto

[TheMidgardWatcher]

But the problem is that I need ssl, as our kafka cluster uses ssl authentication.

[TheMidgardWatcher]

Hi there,

After I've done all the things as it was suggested in https://github.com/redhat-developer/s2i-dotnetcore/issues/152 I've got following message:

An unhandled exception occurred while processing the request.

InvalidOperationException: Failed to load librdkafka at location '/opt/app-root/app/librdkafka.so'. dlerror: '/lib64/libcrypto.so.10: version `OPENSSL_1.0.0' not found (required by /opt/app-root/app/librdkafka.so)'.
Confluent.Kafka.Impl.LibRdKafka.Initialize(string userSpecifiedPath)

I assume that we have OpenSSL v1.0.2k installed in Rhel image that we are using.

Is there any chance to get 'librdkafka.so' compiled against latest version of OpenSSL out of the box?
Or I should go the way where I need to compile it by myself against target platform? And repeat this step after each update of librdkafka.

[mhowlett]

openssl versioning a recurring nightmare for us... As of Confluent.Kafka 0.11.3 we're set up to be able to distribute different versions of librdkafka in a super flexible way (and allow manual loading as a fallback as we're trying to get going here), but we don't include any Rhel builds in librdkafka.redist yet (which is the package referenced by Confluent.Kafka). I don't expect we'll do this for the next release, but it's a pretty popular platform, so I'm sure we'll do it at some point.

i've got some pressing issues elsewhere so can't focus on this just now. But it should definitely be possible to make this work. Make sure you're building librdkafka in the same docker image
/ with the same dependencies that you're deploying to.

[TheMidgardWatcher]

Hello!

I'm glad to report that finally we made it work!

Following your instructions we've built librdkafka.so under Centos7 image and replaced it in my project - and it works!

Thanks for all your suggestions!
I hope you'll find time to release package with built-in lib for Rhel/Centos out of the box.

[edhalsim]

Hi,
I'm new to Linux and Kafka and I'm also trying to get this working on CentOS 7.2. Would someone please tell me exactly what to specify in Library.Load? I'm trying to run the SimpleProducer using: dotnet run ... and I'm getting this error. Thanks for your help.

[TheMidgardWatcher]

Hi @edhalsim

Seems like you are having sane issue as me. So what you need to do:

• Build librdkafka from sources in your environment. Clone git repo of it and follow instructions in repo's readme (find there section 'Build' or something like that)
• Take from the build artifacts file librdkafka.so and put it in any convinient folder (you also can do it like me: include it in your .net project with build action 'Copy always' selected, to be published with your application's files).
• Finally, in your code add line:
  Library.Load(pathToLibrdkafkaSo);
  For instance, if you put 'so' file in you app's root folder, this line would look like this:
  Library.Load("librdkafka.so");

Best regards.

[mhowlett]

thanks @TheMidgardWatcher - beat me to it! one more thing, I think you should also be able to omit the full path to librdkafka if it's installed in a standard location on your system (and you've run ldconfig).

[edhalsim]

@TheMidgardWatcher Thanks for the response. This is what I started doing, but when I ran ./configure I got:
checking for OS or distribution... ok (Linux)
checking for C compiler from CC env... failed
checking for gcc (by command)... failed
checking for clang (by command)... failed
checking for cc (by command)... failed (fail)
checking for C++ compiler from CXX env... failed
checking for C++ compiler (g++)... failed
checking for C++ compiler (clang++)... failed
checking for C++ compiler (c++)... failed (fail)
Apparently I don't have any compiler on this VM. Sorry if this is a dumb question, but how would I get the needed software installed? Thanks.

[freegodo]

I publish use docker, microsoft/aspnetcore:2.0, when docker is run 。report error:Error:
An assembly specified in the application dependencies manifest (Local.deps.json) was not found:
package: 'librdkafka.redist', version: '0.11.3'
path: 'runtimes/linux-x64/native/debian9-librdkafka.so'

what can i do?

[mhowlett]

@freegodo - i'm not sure why that would happen. You could try explicitly referencing librdkafka.redist in your .csproj file in addition to Confluent.Kafka. Let me know if that works.

[mhowlett]

@edhalsim - try yum install gcc gcc-c++. you may need to install other things as well - for those, google 'yum install <whatever is currently failing>'

[edhalsim]

@mhowlett Thanks Matt. I tried:
sudo yum group install "Development Tools"
sudo yum install librdkafka-devel
Then built the library on CentOS and referenced the .so file in my code:
Library.Load("/usr/local/lib/librdkafka.so");
I'm still getting the same error. Also tried copying the .so to the app's bin/Release folder, but I'm still getting the unable to load kernel32.dll.
You mentioned running ldconfig; what's that?

[sfrooster]

This seems to be a "sometimes" issue on osx as well.

On my personal mac, I've had no issues successfully building and running various configurations. I have a new team member however who, no matter what we've tried, can build a netcoreapp2.0 project (which has a project reference to a netstandard2.0 project with a nuget dependency on 0.11.3, the project dependency builds fine as well) but cannot run the netcore app without getting the aforementioned "Unhandled Exception: System.DllNotFoundException: Failed to load the librdkafka native library. at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath)" message.

I'm going to try colocating the actual library (we've already tried installing the redist nuget explicitly, even though it was being installed as a dependency - neither of which worked, or course), but it's baffling why he's having these issues and I am not....

[sfrooster]

I've now added code to Load librdkafka.dylib on both my machine and his and it works fine on mine, and errors - but now with:
Unhandled Exception: System.InvalidOperationException: Failed to load librdkafka at location '/Users/username/.nuget/packages/librdkafka.redist/0.11.3/runtimes/osx-x64/native/librdkafka.dylib'. dlerror: ''.

The file is there, and otool reports the same info for both of our machines.

Any ideas?

[mhowlett]

it's quite likely a (native) shared library dependency problem. librdkafka depends on ssl and sasl libraries on the system and if they are incompatible with what librdkafka expects you'll see what you are seeing. I've not seen anyone have this problem on a mac before. you could try building librdkafa locally and loading that version. There's a way to look at dependencies of a native shared library (that you can use to debug), but I forget how off the top of my head. If you look into this further it'd be great if you paste your findings here, also MacOS versions or anything else relevant... if other people are running into this, please let us know here.

[sfrooster]

@mhowlett "brew install openssl" fixed everything such that he also no longer needs to explicitly call Load.

As for versions, he's on osx 10.13.3, dotnet cli 2.2-preview/2.1, mono 5.8, Confluent.Kafka 0.11.3, building a netstandard2.0 lib used by a netcoreapp2.0 console app.

[mhowlett]

thanks @sfrooster - that's good information. is this a new system or did he get to 10.13.3 via upgrading from an older OS version? we have the capability now to distribute multiple versions of librdkafka per platform, and we'll do that for osx if we see enough people running into this.

[sfrooster]

It's a new mbp, whereas mine is probably 1.5 years old?

[mhowlett]

i'm guessing something's changed recently. thanks again for reporting.

[kuskmen]

Hi guys, I am trying to do exactly what @sfrooster said - building netstandard2.0 lib used by a netcoreapp2.0 console app and it fails with aforementioned error on Windows 10, is there any resolution/solution to this yet?

[mhowlett]

closing since we now include a centos build in librdkafka.redist and this thread is getting a bit long and windy. please open a new issue (reference this one if applicable) for further related issues.

[TheMidgardWatcher]

Thanks @mhowlett , I've checked the lates version already - works perfectly, even on fresh dotnet 2.1 Centos 7 images!

[glazkovalex]

Hi guys, I am trying to do exactly what @sfrooster said - building netstandard2.0 lib used by a netcoreapp2.0 console app and it fails with aforementioned error on Windows 10, is there any resolution/solution to this yet?

I was helped with this code before creating the producer

try
{
    _producer = builder.Build();
}
catch (DllNotFoundException)
{   // Загрузка злоебучей librdkafka.dll
    if (!Library.IsLoaded)
    {
        var pathToLibrd = System.IO.Path.Combine(System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location),
            $"librdkafka\\{(Environment.Is64BitOperatingSystem ? "x64" : "x86")}\\librdkafka.dll");
        _log.Info($"librdkafka is not loaded. Trying to load {pathToLibrd}");
        Library.Load(pathToLibrd);
        _log.Info($"Using librdkafka version: {Library.Version}");
    }
    _producer = builder.Build();
}

[ratri-git]

https://github.com/confluentinc/confluent-kafka-dotnet/issues/778#issuecomment-560225446