Librdkafka: 'No such configuration property: "sasl.kerberos.kinit.cmd"'

[kavyashivakumar]

Description

I have been stuck with this issue for over a day now. can someone kindly help me with this?

I tried various ways but i keep getting
'No such configuration property: "sasl.kerberos.kinit.cmd"'

How to reproduce

confluent-kafka-dotnet (v0.9.5)

var config = new Dictionary {{"sasl.kerberos.kinit.cmd", @"C:\Program Files (x86)\Java\jdk1.8.0_131\bin\kinit.exe -k -t C:Kafka\svc.keytab [email protected]"},{"sasl.kerberos.keytab",@"c:\kafka\svc.keytab" },
{ "bootstrap.servers", "machine.test.group:9092" },{"broker.name","machine.test.group:9092" },
{ "sasl.kerberos.service.name", "kafka" },{"security.protocol","sasl_plaintext"},
{ "sasl.kerberos.principal","[email protected]"},
};
using (var producer = new Producer(config, null, new StringSerializer(Encoding.UTF8)))
{
}

please help. when i execute this command prompt, it works fine. "New Ticket is stored in cache file C:\users...\krb5cc_user

Checklist

Please provide the following information:

• [the one that came with confluent kafka-net v0.9.5 ] librdkafka version (release number or git tag):
• [kafka_2.10-0.10.1.0] Apache Kafka version:
• [no explicit changes ] librdkafka client configuration:
• [windows 7 ] Operating system:
• [yes ] Critical issue

[treziac]

Looking at the code, looks like this parameter is not defined when librdkafka is compiled on visual studio, so the dll in librdkafka.redist embeded in Confluent.Kafka for windows does not support it. I assume there is no library on windows which support this protocol so this was disable
This seems to work on linux on .net core though (just tested the param was correctly loaded) if this is in your scope

[kavyashivakumar]

@treziac ,thank you. I'm willing to give .net core a try. at least we would know that's the issue. I finally managed to get some logging info for librdkafka. here it is. (apologies if anything is missing. i'm quite new to this) (also found out the librdkafka version is 0.9.0)

whilst i try on .net core, i just want to make sure i've sent right parameters to start with. anything that stands out? I'm not quite sure why authentication is failing. if i run this command after kinit, i can send msgs. just not from .net. (also added {"metadata.broker.list","machine.test.group:9092"} in above dictionary)

kafka-console-producer.bat --broker-list .machine.test.group:9092,.machine1.test.group:9092,machine2.test.group:90
2 --topic KAFKA_MM_RISK_RESULT_L0 --security-protocol SASL_PLAINTEXT

some error details below

on this line
var deliveryReport = producer.ProduceAsync(topicName, null, text);

I get, rdkafka#producer-1|METADATA| [thrd:ap]]: Skipping metadata refresh of 1 topic(s): no usable broker

and on
var deliveryReport = producer.ProduceAsync(topicName, null, text);

i get
rdkafka#producer-1|SASL| [thrd:sasl_plaintext://machine.test.group:9092/bootstrap]: sasl_plaintext://machine.test.group:9092/bootstrap: Acquired Kerberos credentials handle (expiry in ....)
rdkafka#producer-1|BROKERFAIL| [thrd:sasl_plaintext://machine.test.group:9092/bootstrap]: sasl_plaintext://machine.test.group:9092/bootstrap: failed err: Local: Authentication failure: rdkafka#producer-1|STATE| [thrd:sasl_plaintext://machine.test.group:9092/bootstrap]: sasl_plaintext://machine.test.group:9092/bootstrap: Broker changed state AUTH -> DOWN

[treziac]

I'm not familar with ssl/sasl, didn't use it yet, you will have to wait for someone else

Concerning librdkafka 0.9.0, where did you get this value? Is this the return of Librdkafka.Library.VersionString?
It should be 0.9.5, if it's something else, there is a problem (and you have to solve this one first) If it's indeed 0.9.0, check if you have any nuget package to rdkafka.internal or other (you should only have Confluent.Kafka, and transitvie reference librdkafka.redist)

bootstrap.servers and metadata.broker.list are aliases, you don't need to use both.

Also when I say .net core, I mean .net core on linux - .net core on windows will not change anything

[kavyashivakumar]

thank you. yes this was return of Librdkafka.Library.VersionString. but however, i just upgraded to 0.9.5. and still have the same problem.

thanks for pointing out the version. OS is a deal breaker for us. I'll wait to see if someone else has any inputs.

[treziac]

That's strange
What linux distrib do you use? Did you have the same return on windows?
Can you copy your obj/projects.assets.json?
Thanks!

[kavyashivakumar]

@treziac i'll check with the infrastructure team. I'm not too sure of the answers regarding linux.
were you able to run kinit command within the code? as param of producer config? the above errors are all without that.

var config = new Dictionary<string, object> {{"log_level",7 },{"debug","broker, protocol, metadata,security"},/*,{"sasl.kerberos.kinit.cmd", @"C:\Program Files (x86)\Java\jdk1.8.0_131\bin\kinit.exe -k -t C:\Kafka\srv_sdfL.keytab [email protected]"},{"sasl.kerberos.keytab",@"c:\kafka\srv.keytab" },*/ { "bootstrap.servers", "brk.machine.test.group:9092" }, { "metadata.broker.list","brk.machine.test.group:9092"}, { "sasl.kerberos.service.name", @"kafka\brk.machine.test.group:9092" },{"security.protocol","sasl_plaintext"}, { "sasl.kerberos.principal","[email protected]"},{"api.version.request","true"} };

but if i try
var config = new Dictionary<string, object> {{"log_level",7 },{"debug","broker, protocol, metadata,security"},,{"sasl.kerberos.kinit.cmd", @"C:\Program Files (x86)\Java\jdk1.8.0_131\bin\kinit.exe -k -t C:\Kafka\srv_.keytab [email protected]"},{"sasl.kerberos.keytab",@"c:\kafka\srv.keytab" }, { "bootstrap.servers", "brk.machine.test.group:9092" }, { "metadata.broker.list","brk.machine.test.group:9092"}, { "sasl.kerberos.service.name", @"kafka\brk.machine.test.group:9092" },{"security.protocol","sasl_plaintext"}, { "sasl.kerberos.principal","[email protected]"},{"api.version.request","true"} };

I get 'No such configuration property: "sasl.kerberos.kinit.cmd"'

[treziac]

yes, sasl.kerberos.kinit.cmd was accepted as a config in producer, but only for .net core on Linux, not on windows (there if failed with this error)

[kavyashivakumar]

@treziac , thank you. do you mean on windows it failed with same error as mine? if so, is this a show stopper?
I kept thinking there's something wrong with the config i have.
is there no way to connect from windows? @edenhill ? any thoughts and help ?

I can connect without kinit as the keytab is cached. the log says it too. but any thoughts @treziac what i'm doing wrong for the error Login failure invalid argument?
rdkafka#producer-1|BROKERFAIL| [thrd:sasl_plaintext://machine.test.group:9092/bootstrap]: sasl_plaintext://machine.test.group:9092/bootstrap: failed err: Local: Authentication failure:

[kavyashivakumar]

@treziac do i've to explicitely build librdkafka dll with some settings? I didn't do that. i used the one that comes with confluent kafka .net.

[rolandyoung]

@kavyashivakumar The first reply from @treziac above explains that

• the dll in librdkafka.redist embeded in Confluent.Kafka for windows does not support SASL
• this might be because there is no library on windows which supports this protocol
• SASL seems to work on linux on .net core, so it it is an issue with the Windows build

Therefore, using .net core on Windows will not change anything regarding SASL support compared to using the dll supplied with Confluent.Kafka.

TL;DR : SASL is not supported in the default Windows build of librdkafka.

(I do not know whether a library is available on Windows which supports the SASL protocol using the API that librdkafka requires.)

[kavyashivakumar]

@rolandyoung thank you for this. I somehow missed the first line from @treziac. thank you both.

[kavyashivakumar]

just out of interest and for knowledge, @rolandyoung, when you say "it is an issue with the windows build,

1. do you mean of librdkafka or windows OS itself?
2. if i rebuild librdkafka.redist from master node, would that help?
3. so what does this mean? if 2 is not possible, is my understanding correct that i might need a completely diff mechanism to connect to Kafka?

my set up is very similar to this qn. just that my mechanism is GSSAPI.

https://github.com/confluentinc/confluent-kafka-dotnet/issues/169

1. but here is the link which explains how to do on windows. https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka

which is why i'm very lost. could you please shed some light.

please accept my apologies in advance. i'm trying to understand if i can get this working.

[rolandyoung]

I'm really sorry, @kavyashivakumar, I don't know any more than I said above. I follow this repo because we use librdkafka in our product and I want to be aware of anything that would influence our choice of when to rebuild against a new release. We don't need librdkafka on Windows and don't yet need SASL, but I could see that you had missed an important point.

To clarify my comment, what I understand @treziac to have said is that the standard build of librdkafka on Windows, as used by the Confluent distribution, does not include support for SASL.

The librdkafka code uses conditional compilation, controlled by build flags, to control whether certain features are available or not. If you build from scratch, the first step is to configure the build. At this point you can choose which features to enable. But if you try to enable features which aren't supported by the libraries available on your build machine, the build will fail. The link you post above shows that it is possible to build librdkafka on Windows with SASL support; I do not know how difficult that is or what prerequisite libraries are required. Maybe you should try it and, if you get stuck, post another issue here.

Alternatively, you could try out Confluent Enterprise (I notice that a 30 day trial is available) and ask their tech support for help. If you are dependent on Confluent.Kafka for a commercial deployment, having access to paid-for support is probably something you should consider in any case.

[kavyashivakumar]

@rolandyoung thank you for a wonderful patient and comprehensive explanation. I'm going to consider confluent enterprise support with my team. thanks again

[edenhill]

SASL GSSAPI is supported on Windows using the Windows' native SSPI interface, not the Kerberos kinit commands.

See this link for more info:
https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka-on-Windows

[kavyashivakumar]

@edenhill thank you Eden, I will give this a try. the only change in our set up is, zookeeper and kafka server are hosted on Linux box. where as kafka producer will be on windows. I'd love some inputs from Horse's mouth. I understand the librdkafka patched with latest confluent doesn't support windows. but if i rebuild librdkafka, will i be able to use it?

[rolandyoung]

I'm sorry, @kavyashivakumar, I may have misled you. The specific parameter "sasl.kerberos.kinit.cmd" is not available on Windows, because it is not used in the Windows native interface. You should not need that parameter if you follow the instructions at the link Magnus gave you above. It may be that the Confluent build already contains SASL support but you need to follow the Windows instructions, not the Debian/Ubuntu instructions at the link you mentioned.

[kavyashivakumar]

@rolandyoung @edenhill I'm sorry i know i've asked this but unless I'm missing something i can't understand one thing. the link you shared explains setting up windows zookeeper. but my setting is different. Zoo keeper and kafka are hosted on Linux. and my client is on windows. I already have keytabs and i know it works because i can use kafka-console-producer.bat file to send data. it just doesn't work on .net with above code.

[nigash]

@kavyashivakumar did you get this working? i have same setup. kafka and zookeeper running on Linux and producer is on Windows(c#). Our environment supports SASL_Plaintext. i am not able to authenticate with confluent client.

[edenhill]

On Windows librdkafka is not using the Kerberos tools, but the Windows SSPI which has Kerberos support.
Please see the howto here:
https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka-on-Windows

Do note that there is a bug that confuses sasl.kerberos.principal (client principal) with sasl.kerberos.service.name (SPN of broker principal).
This means that you currently need to set the SPN of the broker in sasl.kerberos.principal (such as kafka if the full principal will be kafka/brokerhostname@somerealm).
For client principal it will always use the current logon user.

[nigash]

i am bit confuse with
DOMAIN - The Windows domain of the broker HOST. E.g. LONDON. Any casing works.
is broker running on windows??

in my case we have everything setup and running in Linux.

my configuration is.
kafka running on linux under "abc" realm. Already set to accept SASL_PlainText. from windows using Java and keytab files we are able to authenticate.
We are trying same thing with C# client
C# Client will run from windows. all window users have xyz domain eg. xyz\user1

how can i make user1 write to kafka?

[kavyashivakumar]

@nigash
No I couldn’t get it working cuz of all issues mentioned before. @edenhill’s link is helpful if the brokers are on windows. Not for our set up.
I had to use java to get it working.
In parallel see if this link is of any useful for you.
He elaborates quite patiently why it didn’t work for me.

[kavyashivakumar]

Missed the link
https://github.com/SteveSyfuhs/Kerberos.NET/issues/3#issuecomment-336446831

[cleidsonps]

SASL GSSAPI is supported on Windows using the Windows' native SSPI interface, not the Kerberos kinit commands.

See this link for more info:
https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka-on-Windows

@edenhill! Actually, is there any way to use a keytab for authentication on .NET platform?
The service principal are on kerberos MIT server REALM.

Best regards,

[edenhill]

The client will use keytabs on Linux and OSX, and the native SSPI interface o Windows.

[cleidsonps]

But if I need to use a different user than the one that is logged on Windows, is it possible to do it using librdkafka?
In case this is not possible, do you know any other alternatives to solve this situation that I described?

Thanks in advance.

[rnpridgeon]

It is not possible today but some preliminary research suggests that the SSPI API does in fact support this. We will need to dig into it a bit more though before we can say when and where it would live on the roadmap.

[rolandyoung]

It is very difficult to have more than one credential cache on the machine. This is one of the issues cited by detractors of Kerberos.
In the context of Kerberos, "single sign on" tends to mean "you can only be signed on once at any one time". This was a reasonable constraint when it was designed. Its goal was to provide a way to allow a user to enter their credentials just once in order to access all the resources in the domain they were working in.

[meena1977]

any solution to use keytab in .net windows?

[edenhill]

@meena1977 Not that we are aware of, no.