Confluent-kafka-dotnet: Failed to load the librdkafka native library on AWS lambda?

[nkw]

Description

I created a test AWS lambda function using C# and added the reference Confluent.Kafka. It can be run in the local unit test method. However, it got the error when running in AWS.

https://stackoverflow.com/questions/51405466/aws-toolkit-for-visual-studio-bug-aws-c-sharp-lambda-failed-to-load-the-librdka

How to reproduce

1. Create an AWS lambda function using AWS toolkit for Visual Studio
2. Add the reference to Confluent.Kafka
3. Add the following line to the function
   using (var producer = new Producer<Null, string>(config, null, new StringSerializer(Encoding.UTF8))){ ... }

Checklist

Please provide the following information:

• [ ] Confluent.Kafka nuget version: v0.11.5, also tried v0.11.4
• [ ] Apache Kafka version: 2.11-1.1.0
• [ ] Client configuration:
• [ ] Operating system: AWS lambda (testing on Windows 10 works)
• [ ] Provide logs (with "debug" : "..." as necessary in configuration)
  
  14:32:55
  Failed to load the librdkafka native library.
  Failed to load the librdkafka native library.
  
  14:32:56
  at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath) at Confluent.Kafka.Producer..ctor(IEnumerable1 config, Boolean manualPoll, Boolean disableDeliveryReports) at Confluent.Kafka.Producer2..ctor(IEnumerable1 config, ISerializer1 keySerializer, ISerializer1 valueSerializer, Boolean manualPoll, Boolean disableDeliveryReports) at StratsLambda.Function.KProduce[T](D at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath) at Confluent.Kafka.Producer..ctor(IEnumerable1 config, Boolean manualPoll, Boolean disableDeliveryReports)
  at Confluent.Kafka.Producer2..ctor(IEnumerable1 config, ISerializer1 keySerializer, ISerializer1 valueSerializer, Boolean manualPoll, Boolean disableDeliveryReports)
  at StratsLambda.Function.KProduceT in C:\work....\StratsLambda\Function.cs:line 108
  at StratsLambda.Function.FunctionHandler(S3Event evnt, ILambdaContext context) in C:\work....\StratsLambda\Function.cs:line 87
• [ ] Provide broker log excerpts
• [x] Critical issue

[mhowlett]

The error indicates that either librdkafka could not be found, or has dependencies that are not provided by the host system (and consequently failed to load). My guess would be the latter, in which case you could try building a version of librdkafka with limited dependencies and using Library.Load to load it. I have not tried to use the library in this environment - I'd need to do this to comment further. This is a pretty important scenario for the library to work seamlessly and I have opened an internal ticket to track this. Inconveniences with portability is unfortunately a price we pay for wrapping librdkafka instead of developing a fully managed client, but the benefits of this approach, particularly our confidence in reliability, outweigh the negatives.

[nkw]

Thanks, do you know when will the issue be fixed?

[mhowlett]

not imminently unfortunately. ahead of this in priority is locking down the 1.0 api, promoting the 1.0-experimental branch to master and fixing a number of more generally applicable issues.

anyone else running into this, please +1.

[mhowlett]

it's probably worth pointing out that creating a producer is quite expensive - connections to all kafka brokers will be established and there will be a number of requests to the cluster in addition to the produce request (for api compatibility, metadata etc). i think it's likely you're better off doing this via a proxy of some type, where the producer is instantiated in a long running process.

[mhowlett]

I see now that AWS Lambda has the concept of an execution context which is re-used between function invocations. In light of that, the priority of this issue is much higher.

[CasperSkydt]

Is there an estimated time frame for a fix for this issue?

[mhowlett]

it's marked on our roadmap as an important issue to resolve in Q1 next year.

[ChrisZollinger]

I was able to find out what the root of this issue when trying to deploy the Confluent.Kafka to AWS Lambda. I will go through a couple of gotchas and then root of the issue for those trying to deploy the library in this fashion. Based upon my findings I am not sure what the best solution is to the library to resolve the issue.

_Gotcha #1: Your code does not call Confluent.Kafka.Library.Load explicitly._

In order to properly include your dependencies into a zip file for deployment you must run the .net CLI to include the dependencies in the output. If you just zip up your bin folder and then upload it to AWS via the console you will get the error that the shared library librdkafka could not be loaded.

dotnet publish /p:GenerateRuntimeConfigurationFiles=true

This is because librdkafka libraries in the /runtimes/linux-x64/native are not part of the deployment and not on the host machine.

_Gotcha #2: Your code calls Confluent.Kafka.Library.Load explicitly._

This is caused because libdl which is referenced by a DllImport to load the librdkafka library explicitly is not deployed as libdl.so (it is libdl.so.2). It looks like .net core only tries a few different things to take the name in the DllImport and convert it to a native linux file. Appending .2 is not one of them. This will result in the error that the shared library libdl could not be loaded.

I cannot think of a good reason to call .Load explicitly if you publish the lambda function properly.

_Root cause to librdkafka not loading on AWS Lambda_

After putting LD_DEBUG all files on and going through the information I found that the librdkafka dependency for ssl were not installed on the host. Specifically the following files for 0.11.6 version of the library.

libssl.so.1.0.0
libcrypto.so.1.0.0

Since you cannot manage the libraries installed on the host in AWS lambda by calling a package manager. The solution was to include them in the lambda deployment itself. Luckily the folder that your lambda function is deployed to on the host is also in the LD_LIBRARY path. (/var/task)

Therefore you can download binaries from

https://archlinux.pkgs.org/rolling/archlinux-core-x86_64/openssl-1.0-1.0.2.q-1-x86_64.pkg.tar.xz.html

and then include them in the root of your project. Then have your project build copy them to the bin folder by setting the "copy to output directory" property on the project item.

[edenhill]

@ChrisZollinger Those are great instructions! You should write up a blog post about it :)

[atheken]

Based on @ChrisZollinger debugging and instructions, I was able to resolve this issue by ensuring the openssl.redist is installed as a dependency on my project:

dotnet add package openssl.redist

This creates a transitive dependency on that openssl static libs that is resolved when your project is built/restored.

Further, there is a docker container that is useful in rapidly testing this locally:

cd bin/Release
docker run  --rm -v "$PWD:/var/task" lambci/lambda:dotnetcore2.1 <assembly name>::<function namespace and class> <function payload>