Confluent-kafka-dotnet: Using library inside of Linqpad

[Eibwen]

Description

Pretty similar to #409 (Error while loading librdkafka.dll or its dependencies) but inside a different scenario.

When trying to use the library in https://www.linqpad.net/ I get the error about loading librdkafka. This is might be due to Linqpad not understanding native libraries very well? But no work-around for that exists for linqpad.

My understanding of how Linqpad handles nuget references is:

• Downloads a copy of all referenced nugets to:
  %LOCALAPPDATA%\LINQPad\NuGet.FW46\
• With any dependencies there-in in sub folders of that
• Then when you run the script, all the needed dlls are shadow copied to a %temp% folder where the script is complied to
  
  
  
  • there is an advanced setting that will actually copy the dlls, in which case that temp folder is referenced, and if looking into that folder no librdkafka related files are copied
  
  
• So in my case, the error reports its looking for librdkafka.dll in %LOCALAPPDATA%\LINQPad\NuGet.FW46\Confluent.Kafka(Prerelease)\Confluent.Kafka.1.0.0-beta2\lib\net46
• Here my understanding of how linqpad handles references stops

Work around:

• I copied the releases folder from %LOCALAPPDATA%\LINQPad\NuGet.FW46\Confluent.Kafka(Prerelease)\librdkafka.redist.1.0.0-PRE1 (aka the folder that the dependency was downloaded into)
• into %LOCALAPPDATA%\LINQPad\NuGet.FW46\Confluent.Kafka(Prerelease)\Confluent.Kafka.1.0.0-beta2\lib\net46

How to reproduce

You should be able to load this script into a linqpad instance: http://share.linqpad.net/i7v9ej.linq

Checklist

Please provide the following information:

• [x] Confluent.Kafka nuget version: 1.0.0-beta2
• [x] Apache Kafka version: 2.1.0-cp1 (Commit:bda8715f42a1a3db)
• [ ] Client configuration:
• [x] Operating system: Windows (kafka running on docker for windows)
• [ ] Provide logs (with "debug" : "..." as necessary in configuration)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[Eibwen]

The way that #409 was fixed would not be pretty with this case.

The reason I feel it would be very useful to find some sort of fix for this (other than my manual work-around) is because I find linqpad to be a very useful tool for prototyping, testing edge cases, and teaching. As well as being able to share minimal and complete examples with the scripts

It appears that @albahari is Joseph Albahari's account, who might be able to add some details on how Linqpad looks for references

[albahari]

I've taken a look at Confluent.Kafka and I suspect that the problem is due to the way Librdkafka.Initialize loads the librdkafka.dll native module.

LINQPad sets the environment PATH variable to include the folder where the correct native DLL is located just before the query runs.

Librdkafka.Initialize possibly doesn't allow for this - it seems to always specify an absolute path and loads with native DLL the LoadLibraryFlags.LOAD_WITH_ALTERED_SEARCH_PATH flag.

I suspect that it could be fixed by reverting to the normal loading mechanism in the case that it cannot find librdkafka.dll, so that if it's in the OS Environment PATH, it will resolve.

[vcaraulean]

Stumbled upon this problem and found a workaround.

The libdkafka.dll has to be loaded manually. Somewhere at the beginning of the linqpad script add this line:

Library.Load(@"%LOCALAPPDATA%\LINQPad\NuGet.FW46\Confluent.Kafka\librdkafka.redist.1.3.0\runtimes\win-x86\native\librdkafka.dll");

Review the version and pay attention to the bitness of included library, it should be the same as the linqpad bitness.