Confluent-kafka-dotnet: A Self-Contained Deployment for .NET Core is missing librdkafka

[yaakov-h]

Description

When creating a self-contained deployment for a .NET Core application that references Confluent.Kafka, librdkafka is missing from the output directory.

Is this a bug in Confluent.Kafka, librdkafka.redist, or the .NET SDK itself?

How to reproduce

1. Create a new .NET Core 2.0 project.
2. Set the language version in the compiler options to 7.1 or higher.
3. Add a NuGet package reference to Confluent.Kafka
4. Add the following code:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Confluent.Kafka;

namespace KafkaProducer
{
    class Program
    {
        static async Task Main(string[] args)
        {
            try
            {
                var config = new Dictionary<string, object>();
                var p = new Producer(config);
                await p.ProduceAsync("foo", Array.Empty<byte>(), Array.Empty<byte>());
                Console.WriteLine("Sent a message!");
            }
            catch (DllNotFoundException ex)
            {
                Console.WriteLine("Failed to load librdkafka (probably).");
                Console.WriteLine(ex);
            }
            catch (Exception ex)
            {
                Console.WriteLine("Failed to send a message!");
                Console.WriteLine(ex);
            }
        }
    }
}

1. Run:

dotnet publish --configuration Release --runtime win-x64 --output out

1. Run out\KafkaProducer.exe
2. Observe that it crashes as follows:

Failed to load librdkafka (probably).
System.DllNotFoundException: Failed to load the librdkafka native library.
   at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath)
   at Confluent.Kafka.Producer..ctor(IEnumerable`1 config, Boolean manualPoll, Boolean disableDeliveryReports)
   at KafkaProducer.Program.<Main>d__0.MoveNext() in C:\Temp\KafkaProducer\KafkaProducer\Program.cs:line 15

1. Observe that the output directory does not contain librdkafka files, x86 or x64 directories.

C:\Temp\KafkaProducer\KafkaProducer>dir out\*kafka*
 Volume in drive C has no label.
 Volume Serial Number is [REDACTED]

 Directory of C:\Temp\KafkaProducer\KafkaProducer\out

29/03/2018  08:42 PM           110,080 Confluent.Kafka.dll
19/04/2018  10:13 AM            44,099 KafkaProducer.deps.json
19/04/2018  10:13 AM             5,632 KafkaProducer.dll
19/04/2018  10:13 AM            78,336 KafkaProducer.exe
19/04/2018  10:13 AM               592 KafkaProducer.pdb
19/04/2018  10:13 AM                28 KafkaProducer.runtimeconfig.json
               6 File(s)        238,767 bytes
               0 Dir(s)  271,495,278,592 bytes free

C:\Temp\KafkaProducer\KafkaProducer>dir out\x*
 Volume in drive C has no label.
 Volume Serial Number is [REDACTED]

 Directory of C:\Temp\KafkaProducer\KafkaProducer\out

File Not Found

C:\Temp\KafkaProducer\KafkaProducer>

(In a Framework-Dependent Deployment these files are put under runtimes\, but this directory is not present with a Self-Contained Deployment.)

Checklist

Please provide the following information:

• [X] Confluent.Kafka nuget version: 0.11.4
• [ ] Apache Kafka version:
• [ ] Client configuration:
• [X] Operating system: Windows 10 Enterprise v1709 (Build 16299.334)
• [ ] Provide logs (with "debug" : "..." as necessary in configuration)
• [ ] Provide broker log excerpts
• [ ] Critical issue: Unsure

[peterwishart]

I was able to reproduce this - it works if you change --runtime to win7-x64.
Seems to be due to the how the windows native runtimes are declared in the librdkafka.redist package.
I'm not sure if missing the 'portable' RIDs counts as a bug or "nice to have" in librdkafka.

[mhowlett]

thanks @peterwishart
If you omit the --runtime argument, it should work fine (include all runtime versions in the published folder). I've found the nuget format to be a not-very-well-documented-nightmare as soon as you get even a little bit off the beaten path... unlikely we're going to make changes here.

[yaakov-h]

If I omit the --runtime argument, I don't get a self-contained deployment.

Should I file this on https://github.com/dotnet/sdk perhaps?

[mhowlett]

The librdkafa.redist package includes librdkafka builds under the following RIDs: linux-x64, osx-x64, win7-x64, win7-x86 (i.e. not win-x64). Aligning the --runtime parameter with one of these apparently resolved the issue for @peterwishart. I just tested this quickly on my dev machine (macos, .net core 2.1) with a variety of RIDs, including win-x64 and even centos-x64, and they did appear to successfully create an SCD. Perhaps this has been resolved in .NET Core 2.1?

[mhowlett]

It seems we should change the RIDs from win7-x64/win7-x86 to win-x64/win-x86. This is made slightly less trivial because there's also 'win7' related naming in the build folder of the nuget package (relevant to C/C++ apps), and we need to ensure that is correct as well. Marking as enhancement for the moment.

[mhowlett]

we've change the RID from win7-x64/x86 to win-x64/x86 in the 1.0-experimental-9 package.

[RandyLarson]

When targeting framework net461, the runtimes folder is never produced.

dotnet publish --configuration release --framework net461 --runtime win7-x86 --self-contained

This seems to hold for any variant of a dotnet publish that targets --framework net461

• Confluent.Kafka 0.11.5