Confluent-kafka-dotnet: Confluent.Kafka cannot find librdkafka when running from dotnetcore 2.0 in docker

[spadger]

sadkafka.zip

Description

Dotnet core 2 app cannot find librdkafka when running in docker image derived from the microsoft/dotnet base image

How to reproduce

Code attached.

• Ensure you have dotnet core sdk 2 installed
• Extract attached arcive
• dotnet publish sadkafka.csproj (if you want; I already built it)
• docker build --tag sadkafka .

If you were to navigate to /bin/Debug/netcoreapp2.0/publish and exec dotnet sadkafka.dll, everything works fine, however if you run the following

• 'docker run sadkafka'

You should see the following stacktrace

zopadev@ubuntu:/media/sf_git/kafkaschmafka$ docker run sadkafka
Running...
Unhandled Exception: System.TypeInitializationException: The type initializer for 'Confluent.Kafka.Impl.LibRdKafka' threw an exception. ---> System.DllNotFoundException: Unable to load DLL 'librdkafka': The specified module or one of its dependencies could not be found.
 (Exception from HRESULT: 0x8007007E)
   at Confluent.Kafka.Impl.LibRdKafka.NativeMethods.rd_kafka_version()
   at Confluent.Kafka.Impl.LibRdKafka..cctor()
   --- End of inner exception stack trace ---
   at Confluent.Kafka.Impl.LibRdKafka.conf_new()
   at Confluent.Kafka.Impl.SafeConfigHandle.Create()
   at Confluent.Kafka.Producer..ctor(IEnumerable`1 config, Boolean manualPoll, Boolean disableDeliveryReports)
   at Confluent.Kafka.Producer`2..ctor(IEnumerable`1 config, ISerializer`1 keySerializer, ISerializer`1 valueSerializer, Boolean manualPoll, Boolean disableDeliveryReports)
   at sadkafka.Program.Main(String[] args) in D:\git\kafkaschmafka\Program.cs:line 22

Checklist

Please provide the following information:

• [x] Confluent.Kafka nuget version: 0.11.0
• [X] Apache Kafka version: na
• [X] Client configuration: na
• [X] Operating system: ubuntu, windows (works) vs docker: microsoft/dotnet (based on debian:stretch)
• [X] Provide logs (with "debug" : "..." as necessary in configuration) NA
• [X] Provide broker log excerpts NA
• [X] Critical issue Its blocking us (and we have a support contract with you)

[treziac]

Thanks!
It's strange, doing a dotnet publish sadkafka.csproj -r debian.8-x64
and then running the ./sadkafka generated in docker works. I didn't saw anything about a modification of the way runtimes are handled, will investigate when I have time ( @mhowlett don't know when I will have time, I'm really busy for now)

Also, we will need to set netstandard2.0 as a reference to remove some warnings

[spadger]

I ran dotnet publish sadkafka.csproj -r debian.8-x64 as well, and apart from having loads more files in my container's /app folder, nothing changed for me.

[mhowlett]

Thanks for the heads up. We'll definitely get to making sure everything works smoothly with .net core 2 before the next release (which will approximately coincide with the next kafka / confluent platform release).

[mhowlett]

ok, I just did a quick test with the 2.0 docker image. The following warning happens during restore:

/tmp/test/test.csproj : warning NU1701: Package 'librdkafka.redist 0.11.0' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETCoreApp,Version=v2.0'. This package may not be fully compatible with your project.

which looks identical to the issue #291 (on windows).

until we do a new release you'll need to either put the librdkafka shared library (and any dependencies) contained in the librdkafka.redist nuget package alongside Confluent.Kafka.dll or install librdkafka on your system using the confluent packages or by installing from source.

[mhowlett]

I'm guessing this is related to changes made that allow .NET Framework libraries to be referenced from .NET Standard

[treziac]

Well, I did some more testing (I could not replicate what I sais first, I may have mixed something)
I still don't understand - even the text does you paste @mhowlett does not explain it - the runtimes are still here, and librdkafka.so is well copied in output folder. Either a dependencies is missing (but which one?) either dllImport has changed. Will try to look a bit more on this tomorrow as we need this at work

[mhowlett]

yep, it looks like it's a dependency problem:

root@80ee8d9cc728:/tmp/test/bin/Debug/netcoreapp2.0/publish/runtimes/debian-x64/native# ldd librdkafka.so
    linux-vdso.so.1 (0x00007ffff486a000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff8c33aa000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ff8c3190000)
    libcrypto.so.1.0.0 => not found
    libssl.so.1.0.0 => not found
    libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007ff8c2f75000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff8c2d6b000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff8c2b67000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff8c27c8000)
    /lib64/ld-linux-x86-64.so.2 (0x000055df3b3ef000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007ff8c25b1000)

The new microsoft docker image has libssl.so version 1.0.2, and librdkafka requires 1.0.0.

We're actively working on solutions to this and other dependency related issues.

[mhowlett]

(thanks to @edenhill for the above)

[spadger]

Ah, that's good news!

I guess for now we can just perform a native debian .net build and not use the Microsoft base image

[mhowlett]

I think that should work.

[treziac]

@spadger Use microsoft/dotnet:2-runtime-jessie or 2-sdk-jessie in your docker file, this will work
(for reference, this comes from dependencies installed here:
https://github.com/dotnet/dotnet-docker/blob/master/2.0/sdk/jessie/amd64/Dockerfile
https://github.com/dotnet/dotnet-docker/blob/master/2.0/sdk/stretch/amd64/Dockerfile )

[spadger]

Yep @treziac, that works for me; thanks!

[autodidaddict]

I get this exact error message when trying to simply restore or build on Ubuntu. I'm using .NET Core 2.0, and I've even done sudo apt-get install librkkafka1 and that doesn't help. I am unable to use this client library

[spadger]

Did you check the version of libssl? Or downgrade to the Jessie base image?

On 7 Oct 2017 18:40, "Kevin Hoffman" [e-mail] wrote:

I get this exact error message when trying to simply restore or build on
Ubuntu. I'm using .NET Core 2.0, and I've even done sudo apt-get install
librkkafka1 and that doesn't help. I am unable to use this client library

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/confluentinc/confluent-kafka-dotnet/issues/294#issuecomment-334953100,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANOrI9FzjAsQgAWUcg1U1HZbJCwr2g9ks5sp7ePgaJpZM4O41Ez
.

[mhowlett]

@autodidaddict - installing the debian package unfortunately won't help for two reasons: the version of librdkafka within it depends on libssl1.0.0 (not libssl1.0.2) and the dotnet runtime will preferentially use the librdkafka.so from the librdkafka.redist nuget package (which is a dependency of Confluent.Kafka).

You have two options at this time: 1. run within a docker container that has the older version of libssl (as described in previous comments). 2. compile a suitable librdkafka from source and overwrite the binary supplied by librdkafka.redist with this compiled version (location: [project dir]/bin/[Debug|Release]/netcoreapp2.0/publish/runtimes/debian-x64/native).

Example steps to do this (untested):

apt-get update
apt-get install gcc
apt-get install g++
apt-get install libssl-dev
apt-get install make
cd ~
wget https://github.com/edenhill/librdkafka/archive/v0.11.0.tar.gz
tar xvf v0.11.0.tar.gz
cd librdkafka-0.11.0
./configure
make
make install
cp /usr/local/lib/librdkafka.so [project dir]/bin/Debug/netcoreapp2.0/publish/runtimes/debian-x64/native

We are in the process of doing a point release for librdkafka (in fact Confluent.Kafka 0.11.1 RC1 is out on nuget, though this issue is not yet resolved). We will address this in that release.

[humbledeveloper43]

Hello,

Any updates for this issue? I'm dealing with exactly this problem on dotnet core 2.0 on fedora 25 linux (not docker).

How can I solve it?

[mhowlett]

Currently working on it. I'm re-packaging librdkafka.redist and making some changes to the dotnet library to make native library loading more flexible. I thought it would be done by now but it's turned out to be quite difficult.

[humbledeveloper43]

@mhowlett OK, Good luck! But I must say that I'm struggling with other nuget kafka libraries at the same time to catch my deadline. So whenever you complete please update the ticket and library so I can return back to use confluent-kafka library. Also if you have any beta updates I can be the tester.

Thanks

[GarrettClyde]

Hey guys, I was getting this error with microsoft/dotnet:2-runtime. Switching to the microsoft/dotnet:2-runtime-jessie image suggested by @treziac, seemed to solve the problem.

Now, a couple weeks later, I just updated another project to dotnet core 2.0, and even with the 2-runtime-jessie image, I'm seeing this issue.

I got a shell into the container and ran a search for the libraries listed in one of @mhowlett's posts above and these are the paths I found:

    /lib/x86_64-linux-gnu/libpthread.so.0
    /lib/x86_64-linux-gnu/libz.so.1
    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
    /usr/lib/x86_64-linux-gnu/libsasl2.so.2
    /lib/x86_64-linux-gnu/librt.so.1
    /lib/x86_64-linux-gnu/libdl.so.2
    /lib64/ld-linux-x86-64.so.2
    /lib/x86_64-linux-gnu/libresolv.so.2

[AndreSteenbergen]

Don't forget when you use 0.11.1 nuget package the librdkafka is in another nuget package, you have to manually add that nuget package. It is easy to see BTW, dotnet publish will throw warnings about nog having the correct compile target. If you don't get them, the redist package is not included ...

[AndreSteenbergen]

Contents of a working nuget situatation (docker with jessie)

 <ItemGroup>
    <PackageReference Include="Confluent.Kafka" Version="0.11.1" />
    <PackageReference Include="Newtonsoft.Json" Version="10.0.3" />
    <PackageReference Include="librdkafka.redist" Version="0.11.1" />
  </ItemGroup>

[humbledeveloper43]

Hello @mhowlett,

Any updates for this issue? (Dotnet Core 2.0 and confluent-kafka-dotnet usage on linux)

[TheFireCookie]

As a workaround, we installed the library librdkafka-devel on CentOS ( sudo yum install librdkafka-devel ) and it works now. Just commenting to help others ;)

[edenhill]

@TheFireCookie That is actually the preferred approach since it will provide the proper dependencies 👍 You should be fine with the librdkafka1 or librdkafka package though, -devel not required for runtime.

[GarrettClyde]

Thanks @AndreSteenbergen
An explicit reference to librdkafka.redist wasn't my specific issue, but your comment got me looking in the right place. It looks like this package behaves differently depending on the runtime identifier used in the dotnet publish command.

I was having issues with my Kubernetes cluster and the dotnet debian.8-x64 RID so I changed it to the new linux-x64 RID.

When publishing projects using either debian-x64 or debian.8-x64, I'm seeing

    out/
    ├── ...
    ├── librdkafka++.so
    ├── librdkafka.so
    └── x64/
        ├── librdkafka.dll
        ├── librdkafka.lib
        ├── librdkafkacpp.dll
        ├── librdkafkacpp.lib
        └── ...

When publishing using --runtime linux-x64, I'm not seeing the .so files

    out/
    ├── ...
    └── x64/
        ├── librdkafka.dll
        ├── librdkafka.lib
        ├── librdkafkacpp.dll
        ├── librdkafkacpp.lib
        └── ...

For anyone else running into this issue, changing my RID to debian-x64 allowed my container to run happily in kops-managed Kubernetes cluster with the confluent-kafka-dotnet library.

[mhowlett]

Thanks @GarrettClyde. I assume you must have also installed librdkafka by some other means?

@humbledeveloper43 - I'm in the middle of completely re-vamping the way native library loading works and re-packaging librdkafka. Yep, same status as 4 days ago, unfortunately not done yet.

[bigmacd]

I am using dotnet core 2.0 (2.0.3) and including Confluent.Kafka version 0.11.2 and librdkafka.redist 0.11.1.

On windows it runs fine. When deployed to a Docker container (FROM microsoft/aspnetcore-build-nightly), it throws the following exception when I try to publish a message:

The type initializer for 'Confluent.Kafka.Impl.LibRdKafka' threw an exception."

What would anyone suggest as the solution. I have read through all the previous comments here and there seems to be a mixed bag of solutions. The big difference is we build our app in the container (hence aspnetcore-build) and we use nightly because we use the bleeding-edge System.Data.Odbc package.

Any help will be much appreciated.

Could it be related to this?:

root@72f0e7dc6151:~/.nuget/packages/librdkafka.redist/0.11.1/runtimes/debian-x64/native# ldd librdkafka.so
linux-vdso.so.1 (0x00007ffd7439a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6622932000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6622718000)
libcrypto.so.1.0.0 => not found
libssl.so.1.0.0 => not found
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f66224fd000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f66222f3000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f66220ef000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6621d50000)
/lib64/ld-linux-x86-64.so.2 (0x00005576ac3c4000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f6621b39000)

libssl and libcrypto are not found.

Also:

root@72f0e7dc6151:~/.nuget/packages/librdkafka.redist/0.11.1/runtimes/debian-x64/native# ldd librdkafka++.so
linux-vdso.so.1 (0x00007ffd1054c000)
librdkafka.so.1 => not found
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f60a449c000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f60a4285000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f60a3ee6000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f60a3be2000)
/lib64/ld-linux-x86-64.so.2 (0x0000564e68f0f000)

Thanks again for looking.

[mhowlett]

incompatibility with the dotnet core 2 docker image is resolved in CI build: 0.11.2-ci-164 and will be included in the 0.11.3 release which will most likley happen later this week.

[mhowlett]

0.11.3 is out and is compatible with microsoft's dotnet core 2.0 docker image.

[kchaithanya]

@mhowlett we are building using dotnet core 2.1 docker image (FROM microsoft/dotnet:2.1-runtime), successfully build the image but while running the image we are seeing below error.

Error:
An assembly specified in the application dependencies manifest (Distribution.Service.deps.json) was not found:
package: 'librdkafka.redist', version: '0.11.4'
path: 'runtimes/linux-x64/native/debian9-librdkafka.so'

This same build works well when deployed as windows service. Could you please help/suggest.

[mhowlett]

without researching, I'm not sure what the issue is. i'd either try using to the dotnet:2.1-sdk docker image instead, or switching to the 1.0-experimental-3 Confluent.Kafka nuget package. I've been using the 1.0-experimental-3 build myself with the dotnet:2.1-sdk docker image without any problems.

[kchaithanya]

Thanks @mhowlett.. will modify with your recommendation.

[kchaithanya]

@mhowlett Did some more debugging and I made it worked by adding build step inside Docker file as below

FROM microsoft/dotnet:2.1-sdk AS build-env
WORKDIR /app
COPY *.csproj ./
RUN dotnet restore
COPY . ./
RUN dotnet publish -c Release -o out
FROM microsoft/dotnet:2.1-runtime
WORKDIR /app
COPY --from=build-env /app/out ./
ENTRYPOINT ["dotnet", "KafkaPublisher.dll"]

Where as,
in earlier case I used below Docker file which has only runtime and build will be done separately and published using dotnet publish into build_out path.

FROM microsoft/dotnet:2.1-runtime
WORKDIR /app
COPY build_output/. /app/
ENTRYPOINT ["dotnet", "KafkaPublisher.dll"]

Could you please help why doing build/publish out side docker file is not able to identity the librdkafka?

Thanks
Krishna.