Confluent-kafka-dotnet: Unable to load 'librdkafka' in CentOS

[suganthkumar]

I have used the latest 0.9.5.RC4 and it runs properly on windows. but when i try to run it on CentOS it throws

Unhandled Exception: System.TypeInitializationException: The type initializer for 'Confluent.Kafka.Impl.LibRdKafka' threw an exception. ---> System.DllNotFoundException: Unable to load DLL 'librdkafka': The specified module could not be found.
(Exception from HRESULT: 0x8007007E)
at Confluent.Kafka.Impl.LibRdKafka.NativeMethods.rd_kafka_version()
at Confluent.Kafka.Impl.LibRdKafka.version()
at Confluent.Kafka.Impl.LibRdKafka..cctor()
--- End of inner exception stack trace ---
at Confluent.Kafka.Impl.LibRdKafka.conf_new()
at Confluent.Kafka.Impl.SafeConfigHandle.Create()
at Confluent.Kafka.Consumer..ctor(IEnumerable1 config) at Confluent.Kafka.Consumer2..ctor(IEnumerable1 config, IDeserializer1 keyDeserializer, IDeserializer`1 valueDeserializer)

I have manually added the rhel-x64 links in the deps.json file, but i end up in having version mismatch error.

Unhandled Exception: System.TypeInitializationException: The type initializer for 'Confluent.Kafka.Impl.LibRdKafka' threw an exception. ---> System.IO.FileLoadException: Invalid librdkafka version 904c9, expected at least 904ff
at Confluent.Kafka.Impl.LibRdKafka..cctor()
--- End of inner exception stack trace ---
at Confluent.Kafka.Impl.LibRdKafka.conf_new()
at Confluent.Kafka.Impl.SafeConfigHandle.Create()
at Confluent.Kafka.Consumer..ctor(IEnumerable1 config) at Confluent.Kafka.Consumer2..ctor(IEnumerable1 config, IDeserializer1 keyDeserializer, IDeserializer`1 valueDeserializer)

i think the librdkafka redist is not updating the rhel-x64 version of librdkafka libraries.

[mhowlett]

right, so there are two problems - 1. on CentOS the native libraries are not being deployed correctly. when that is resolved manually 2. the version number for rhel-x64 libraries in the package apparently indicates they are in fact older than 0.9.5.

I have not looked into problem 1 (which we need to be fixed before calling this a release). For problem 2, as a potential quick fix I changed the check in Confluent.Kafka to allow librdkafka version 0.9.4 (note: the only critical fix in 0.9.5 is for the CPU issue for windows users). I put the corresponding nuget package from appVeyor on nuget.org: 0.9.5-RC5-ci-207 (which is hidden but accessible).

[edenhill]

Ive verified that the linux libraries included in librdkafka.redist.0.9.5 indeed report version 0.9.5 (0x000905ff)

[mhowlett]

thanks @edenhill

On CentOS, I see that the Windows .dlls are being copied into the x86 and x64 directories. I also see this was the case in 0.9.4.

@suganthkumar we obviously want to fix this. but something to consider right now: if docker is an option, this issue does not occur when running inside the microsoft/dotnet docker images.

[mhowlett]

This is not a regression (was broken in 0.9.4 as well), so we've decided to postpone fixing until 0.9.6 as 0.9.5 contains a critical fix for windows users that we want to get out now.

[treziac]

@mhowlett
Strange, the dll should only be copied for net project, or nuget works differently on centOs.
(didn't test the rhel package on my side, only the debian ones)

[mhowlett]

Ideally we should set up an automated test for each supported platform. It's time consuming to manually do a basic test on all platforms prior to release and easy to cut corners..

[suganthkumar]

i did a full compile on CentOS for the librdkafka library, and once the new library is deployed, its working properly. so its seems the only issue is nuget is not deploying the right native libraries.

[fvenezia]

Do you have a workaround until next release?

[mhowlett]

Confluent.Kafka.dll just needs to be able to find librdkafka. install it from source (not hard) which will put it in /usr/local/lib or put it in the same directory as Confluent.Kafka.dll.

[zsebastian]

I have this issue too, and I've had it for as long as I've used this package. So far I've been able to fix it by simply installing the librdkafka-devel and librdkafka1 from the confluent package on yum (following these instructions: http://docs.confluent.io/current/installation.html). However, I've tried updating to 0.9.5 of this project and that hasn't worked out. This is what I got this exception: System.IO.FileLoadException: Invalid librdkafka version 902c9, expected at least 903ff

I've not had time to compile the dll from source, but I can do that later and report back if it helps. The latest RdKafka.Internal.librdkafka is 0.9.2-ci28. Is this correct? Is it not supposed to follow the versions of the other packages (i.e. I would naively expect 0.9.5).

[mhowlett]

for CentOS i'd recommend getting the librdkafka source git clone https://github.com/edenhill/librdkafka.git then follow the build/installation instructions to build from source (it's not hard).

the nuget package RdKafka.Internal.librdkafka is no longer relevant - we reference librdkafka.redist.

[a2glukhov]

Personally, I can confirm that this solution (recompile library directly from CentOS) works for .net core projects and solves the issue.

[treziac]

Thanks for confirming - this should also be resolved in next nuget package
We may add doozer CI for testing on centos soon

[yang-xiaodong]

Version 0.11.0

I also have this problem on CentOS, I've solved it through the following command :

sudo yum install librdkafka-devel

[mhowlett]

centos7 compatible builds of librdkafka are now included in librdkafka.redist - manual install of librdkafka is no longer required on this platform.