Hi,
I've been trying to apply the SASL Kerberos authentication with the use of a keytab (principal name & encrypted password) using the Confluent.Kafka library (version 0.11.2)
I am struggling to pass a keytab file from Kafka .NET client (residing in Window server) to the Kafka broker (residing in Linux server).
I setup the properties for Kafka Broker (server.properties) as below
listeners=SASL_PLAINTEXT://:9092
security.inter.broker.protocol= SASL_PLAINTEXT
security.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=GSSAPI
Below is my "Producer" code (in .NET)
var config = new Dictionary
{ "bootstrap.servers", bootstrapServer },
{ "api.version.request", true },
{ "group.id", groupId },
{ "sasl.mechanisms", "GSSAPI" },
{ "security.protocol", "SASL_PLAINTEXT" },
{ "sasl.kerberos.service.name", "kafka" },
{ "sasl.kerberos.principal", principalName }
};
using (var producer = new Confluent.Kafka.Producer(config))
{
producer.ProduceAsync(topic, null, new StringSerializer(Encoding.UTF8).Serialize(message));
producer.Flush(TimeSpan.FromSeconds(10));
}
Questions:
Thank you very much.
-Hoang
You don't actually use a keytab file for a client running on a windows machine. librdkafka uses a different library for SASL on windows machines. There's also a bit if a bug in the code for windows. It doesn't actually use sasl.kerberos.service.name at all, instead it uses sasl.kerberos.principal. Underneath the covers, the SASL library sends the principal executing your client as the identity authenticated with Kafka rather than using a keytab file. We haven't gone to Prod with this yet but using SASL_SSL I believe this is the config we used.
__Producer__
var config = new Dictionary<string, object>
{
{ "client.id", "clientid"},
{ "api.version.request", "true" },
{ "security.protocol","sasl_ssl"},
{ "ssl.ca.location","pemfilelocation.pem"},
{ "sasl.mechanisms", "GSSAPI"},
{ "sasl.kerberos.principal", "kafka"},
{ "bootstrap.servers", brokerList}
};
__Consumer__
var config = new Dictionary<string, object>
{
{ "group.id", consumerGroup },
{ "client.id", "clientid"},
{ "enable.auto.commit", false },
{ "statistics.interval.ms", 60000 },
{ "bootstrap.servers", brokerList },
{ "security.protocol","sasl_ssl"},
{ "ssl.ca.location","pemfilelocation.pem"},
{ "sasl.mechanisms", "GSSAPI"},
{ "sasl.kerberos.principal", "kafka"},
{ "api.version.request", "true" },
{ "default.topic.config", new Dictionary<string, object>()
{
{ "auto.offset.reset", "smallest" }
}
}
};
thanks for that @petersonnek. IIRC, the configuration bug you mention has been resolved and should no longer be a problem in version 0.11.3 which i'm about to do a RC for.
Thank you @petersonnek!
Does that mean I need to create a broker keystore for the pem cert (i.e. pemfilelocation.pem), and configure the broker's configuration file "server.properties"?
If so, how can I configure these to the configuration file, "server.properties" ?
Also, I want to come back to my initial question.
How do I setup Kerberos for a Kafka .NET client (instead of your suggestion to use TLS/SSL)?
We're using SASL_SSL not SASL_PLAINTEXT. You don't need { "ssl.ca.location","pemfilelocation.pem"} for your situation.
I'm posting our security confing in server.properties below for SASL_SSL. We are transition away from PLAINTEXT. The SSL and SASL_PLAINTEXT listeners are enabled because it helped us debug getting SASL_SSL working. You can ignore all of the PLAINTEXT, SSL, and SASL_SSL config for your SASL_PLAINTEXT scenario. Refer to the Kafka documentation for more details https://kafka.apache.org/documentation/#security
In addition to the server.properties, per the documentation, make sure you have a dedicated broker ID per broker for authentication. If you're running more than one broker, you need a shared ID for zookeeper authentication assuming you've set zookeeper.set.acl=true in server.properties and requireClientAuthScheme=sasl in zookeeper.properties.
Finally, make sure you have JAAS configuration files for each of your brokers that are passed in as JVM parameters at the start of Kafka.
Assuming your machines (windows & linux) are on the same realm (right now windows + librdkafka don't support multiple realms) and you are using kafka.security.auth.SimpleAclAuthorizer out-of-the-box, when you add ACLs for a user, leave off the realm.
Works out-of-box
kafka-acls --allow-principal User:[UserID] --authorizer-properties zookeeper.connect=[zookeeperHost:port] --add --producer --topic [topic]
Doesn't work out-of-box
kafka-acls --allow-principal User:[UserID]@REALM.COM --authorizer-properties zookeeper.connect=[zookeeperHost:port] --add--producer --topic [topic]
With some tweaking that I didn't look into, you can get set up the authorizer to include the realm. We didn't need the realm in our situation.
__Security config in server.properties__
listeners=PLAINTEXT://FQDNhostname:9092,SSL://FQDNhostname:9093,SASL_SSL://FQDNhostname:9094,SASL_PLAINTEXT://FQDNhostname:9095
advertised.listeners=PLAINTEXT://FQDNhostname:9092,SSL://FQDNhostname:9093,SASL_SSL://FQDNhostname:9094,SASL_PLAINTEXT://FQDNhostname:9095
# SSL configs
ssl.keystore.location=brokercertkeystorepath.jks
ssl.keystore.password=password
ssl.key.password=passord
# root truststore
ssl.truststore.location=CAkeystorepath.jks
ssl.truststore.password=passord
ssl.client.auth=none
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
# Security configs
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=GSSAPI
zookeeper.set.acl=true
sasl.enabled.mechanisms=GSSAPI,PLAIN
sasl.kerberos.service.name=kafka
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
__kafka_server_jaas.conf__
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/pathtokeytab/hostname-kafka.ktf"
principal="kafka/[email protected]";
};
//Zookeeper client authentication
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/pathtokeytab/kafka-zookeeper-client.ktf"
principal="[email protected]";
};
I am trying the same , .Net producer on windows with SASL_PLAINTEXT and broker on unix. What kind of settings I need to do in windows to make it work.
Is it mandatory to use the keytab or can you use the Ticket Cache (useTicketCache = true) as used in Java?
Using the configuration below I can not connect to kafka while the example kafka-console-consumer works correctly
var config = new Dictionary<string, object>
{
{ "group.id", consumerGroup },
{ "client.id", clientId},
{ "statistics.interval.ms", 60000 },
{ "bootstrap.servers", brokerList },
{ "security.protocol","sasl_plaintext"},
{ "sasl.mechanisms", "GSSAPI"},
{ "sasl.kerberos.principal", "kafka"},
{ "api.version.request", "true" }
};
these are lines of the debugged error
7|2018-02-08 11:42:38.689|clientid#consumer-1|SASL| [thrd:sasl_plaintext://192.168.111.2:9092/bootstrap]: sasl_plaintext://192.168.111.2:9092/bootstrap: Initializing SASL client: service name kafka, hostname 192.168.111.2, mechanisms GSSAPI, provider Win32 SSPI
7|2018-02-08 11:42:38.692|clientid#consumer-1|SASL| [thrd:sasl_plaintext://192.168.111.2:9092/bootstrap]: sasl_plaintext://192.168.111.2:9092/bootstrap: Acquired Kerberos credentials handle (expiry in 2147483455.928712703s)
7|2018-02-08 11:42:38.692|clientid#consumer-1|BROKERFAIL| [thrd:sasl_plaintext://192.168.111.2:9092/bootstrap]: sasl_plaintext://192.168.111.2:9092/bootstrap: failed: err: Local: Authentication failure: (errno: Invalid argument)
7|2018-02-08 11:42:38.697|clientid#consumer-1|STATE| [thrd:sasl_plaintext://192.168.111.2:9092/bootstrap]: sasl_plaintext://192.168.111.2:9092/bootstrap: Broker changed state AUTH -> DOWN
7|2018-02-08 11:42:42.000|clientid#consumer-1|BROADCAST| [thrd:sasl_plaintext://192.168.111.2:9092/bootstrap]: Broadcasting state change
Microsoft Windows [Versione 10.0.16299.192]
.net 4.6
Kafka 0.11.0
Scala 2.11
Confluent's .NET Client for Apache Kafka (Current Version)
@etzellava To kind of answer your initial question, the .NET consumer, doesn't use a keytab file. It uses the principal of the process running the consumer.
One thing I noticed, if you're using the latest Confluent Kafka .Net client, then you need to use
{ "sasl.kerberos.service.name", "kafka"} rather than { "sasl.kerberos.principal", "kafka"}. That was a bug in a previous version that was fixed. The config I initially posted was from an older version of the client.
I also notice that you're using port 9092, which is typically used for PLAINTEXT. It's quite possible you set up your cluster to use 9092 for SASL_PLAINTEXT but I thought I'd point that out too.
@pradipkumar1948 If you read through this thread I outlined most of you need both from a kafka and a windows perspective. Are there specific details you need help with?
@petersonnek i changed configuration
var config = new Dictionary<string, object>
{
{ "group.id", "simple-csharp-consumer" },
{ "client.id", "clientid"},
{ "bootstrap.servers", brokerList },
{ "security.protocol","SASL_PLAINTEXT"},
{ "sasl.mechanisms", "GSSAPI"},
{ "debug","all"},
{ "sasl.kerberos.service.name", "kafka"} ,
{ "api.version.request", "true" },
{ "default.topic.config", new Dictionary<string, object>()
{
{ "auto.offset.reset", "smallest" }
}
}
};
But the result didn't change
it look like SSPI couldn't find the user kerberos token.
Log in attach file
KafkaClientNet.txt
I followed this link
https://maprdocs.mapr.com/52/Impala/configuring_kerberos_authentication_for_windows.html
and kafka-console-consumer(confluent-4.0.0 bin) work without problem.
Log in attach file
KafkaClientJava.txt
Attached configuration file
What about dot net core? That can run either on Windows or Linux. What api is used to get the current user here?
You're correct, if you are choosing to run .NET core on Linux, then you would use a keytab.
librdkafka references/uses different libraries for kerberos depending on the platform you run the .NET consumer
Thanks Peter, we are still having issues from Windows Clients. Let's assume that the dot net core process in Windows is running under a service account, for example 'svcKafkaTestAccount'. This account exists in AD. We have generated a keytab file for it out of AD. Would we set the config on the client as:
{ "sasl.kerberos.service.name", "svcKafkaTestAccount"}
? Is that all we'd have to do?
And deploy the keytab on the Linux server running Kafka? Do we also need to set the acls?
Thx
For { "sasl.kerberos.service.name", "svcKafkaTestAccount"} you most likely need to set the value to kafka, not svcKafkaTestAccount. It depends on the value you've set for sasl.kerberos.service.name in your broker server.properties file. The sasl.kerberos.service.name, which you've associated to a AD principal via the KafkaServer configuration in your .jaas file, is what you are requesting access to with svcKafkaTestAccount via kerberos. Kerberos is hard and I don't claim to be an expert but that's the gist of it.
Given this scenario, you should not need a svcKafkaTestAccount keytab at all. Not on the Windows box or the Linux Kafka brokers. You only need keytab(s) on the Linux Kafka brokers for the account(s) used in the .jaas file passed in to -Djava.security.auth.login.config at start up.
You will need to add Kafka ACLs on the broker for svcKafkaTestAccount.
One more note, the library used by confluent-kafka-dotnet for Kerberos on Windows transparently leverages the principal your client is running under. In this case, it is
the dot net core process in Windows running under a service account
Thanks Peter, I was aware of the last fact.
@petersonnek , what would the config look look if the .net core client was hosted on linux? Would we use a keytab and if yes, how do we tell librdkafka to use the Keytab?
Thx
Yes, you'll need a keytab file to run on Linux. I haven't done this myself but I'll give you my best guess.
The config should be pretty similar to hosting on Windows. Based on the sasl.kerberos.kinit.cmd property in the librdkafka documentation, I think you need to set the following in your consumer/producer config in addition to the other config set for Windows.
sasl.kerberos.service.name - most likely kafkasasl.kerberos.keytab - path to your keytab filesasl.kerberos.principal - the principal name in your keytabI'm not 100% sure but you might need to compile your .NET Core app on a Linux machine in order to have the proper librdkafka assembly referenced by your code. If you're successful, post an update about the compilation.
thanks again @petersonnek for helping out here. closing in favor of an 'enhancement' issue to create an example / walkthrough of this, and other security related config.
Hi,
I am trying to create a .NET client which connects with Kafka to post a message. Kafka is running in Linux with Kerberos authentication configured. Java solution works as expected, but .NET code returns _rdkafka#producer-1|BROKERFAIL| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: failed: err: Local: Authentication failure: (errno: Invalid argument)
Error trace is below. I dont have any .NET expert with experience in this area. Appreciate your help.
Start -- Sending a Sample Message to Kafka Topic
7|2018-10-23 15:57:04.197|rdkafka#producer-1|SASL| [thrd:app]: Selected provider Win32 SSPI for SASL mechanism GSSAPI
7|2018-10-23 15:57:04.342|rdkafka#producer-1|BROKER| [thrd:app]: sasl_plaintext://myserver:port/bootstrap: Added new broker with NodeId -1
7|2018-10-23 15:57:04.342|rdkafka#producer-1|INIT| [thrd:app]: librdkafka v0.11.6 (0xb06ff) rdkafka#producer-1 initialized (builtin.features 0xffff, debug 0x282)
7|2018-10-23 15:57:04.363|rdkafka#producer-1|BRKMAIN| [thrd::0/internal]: :0/internal: Enter main broker thread
7|2018-10-23 15:57:04.371|rdkafka#producer-1|STATE| [thrd::0/internal]: :0/internal: Broker changed state INIT -> UP
7|2018-10-23 15:57:04.363|rdkafka#producer-1|BRKMAIN| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Enter main broker thread
7|2018-10-23 15:57:04.373|rdkafka#producer-1|CONNECT| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: broker in state INIT connecting
7|2018-10-23 15:57:04.520|rdkafka#producer-1|CONNECT| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Connecting to ipv4#myserverIP:port (sasl_plaintext) with socket 1348
7|2018-10-23 15:57:04.522|rdkafka#producer-1|STATE| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Broker changed state INIT -> CONNECT
7|2018-10-23 15:57:04.578|rdkafka#producer-1|CONNECT| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Connected to ipv4#myserverIP:port
7|2018-10-23 15:57:04.578|rdkafka#producer-1|CONNECTED| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Connected (#1)
7|2018-10-23 15:57:04.578|rdkafka#producer-1|FEATURE| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
7|2018-10-23 15:57:04.578|rdkafka#producer-1|STATE| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
7|2018-10-23 15:57:04.578|rdkafka#producer-1|SEND| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1)
7|2018-10-23 15:57:04.638|rdkafka#producer-1|RECV| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Received ApiVersionResponse (v0, 120 bytes, CorrId 1, rtt 59.55ms)
7|2018-10-23 15:57:04.640|rdkafka#producer-1|FEATURE| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4
7|2018-10-23 15:57:04.641|rdkafka#producer-1|AUTH| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Auth in state APIVERSION_QUERY (handshake supported)
7|2018-10-23 15:57:04.641|rdkafka#producer-1|STATE| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Broker changed state APIVERSION_QUERY -> AUTH_HANDSHAKE
7|2018-10-23 15:57:04.641|rdkafka#producer-1|SEND| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Sent SaslHandshakeRequest (v0, 29 bytes @ 0, CorrId 2)
7|2018-10-23 15:57:04.697|rdkafka#producer-1|RECV| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Received SaslHandshakeResponse (v0, 14 bytes, CorrId 2, rtt 55.69ms)
7|2018-10-23 15:57:04.698|rdkafka#producer-1|SASLMECHS| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Broker supported SASL mechanisms: GSSAPI
7|2018-10-23 15:57:04.699|rdkafka#producer-1|AUTH| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Auth in state AUTH_HANDSHAKE (handshake supported)
7|2018-10-23 15:57:04.699|rdkafka#producer-1|STATE| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Broker changed state AUTH_HANDSHAKE -> AUTH
7|2018-10-23 15:57:04.700|rdkafka#producer-1|SASL| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Initializing SASL client: service name kafka, hostname myserver, mechanisms GSSAPI, provider Win32 SSPI
7|2018-10-23 15:57:04.717|rdkafka#producer-1|SASL| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Acquired Kerberos credentials handle (expiry in 2147483404.-322922497s)
7|2018-10-23 15:57:04.884|rdkafka#producer-1|BROKERFAIL| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: failed: err: Local: Authentication failure: (errno: Invalid argument)
7|2018-10-23 15:57:04.885|rdkafka#producer-1|STATE| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Broker changed state AUTH -> DOWN
7|2018-10-23 15:57:05.888|rdkafka#producer-1|CONNECT| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: broker in state DOWN connecting
7|2018-10-23 15:57:05.909|rdkafka#producer-1|CONNECT| [thrd:sasl_plaintext://myserver:port/bootstra]: sasl_plaintext://myserver:port/bootstrap: Connecting to ipv4#myserverIP:port (sasl_plaintext) with socket 1348
Below is my code. Appreciate any help here...
using System;
using System.Collections.Generic;
using System.Text;
using Confluent.Kafka;
using Confluent.Kafka.Serialization;
namespace KafkaDotnetTest
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Start -- Sending a Sample Message to Kafka Topic");
//send message to kafka
SendMessageUsingConfluentKafka("");
Console.WriteLine("Complete -- Sending a Sample Message to Kafka Topic");
}
public static void SendMessageUsingConfluentKafka(string msg)
{
string brokerList = "servername.domainname.com:6667";
string topicName = "mytopicname";
var config = new Dictionary<string, object>
{
//{ "client.id", "clientid"},
{ "api.version.request", "true" },
{ "security.protocol","SASL_PLAINTEXT"},
//{ "ssl.ca.location","pemfilelocation.pem"},
{ "sasl.mechanisms", "GSSAPI"},
{ "sasl.kerberos.service.name", "kafka" },
{ "debug", "security,broker,protocol"},
{ "sasl.kerberos.principal", "[email protected]"},
{ "bootstrap.servers", brokerList}
};
using (var producer = new Producer<string, string>(config, new StringSerializer(Encoding.UTF8), new StringSerializer(Encoding.UTF8)))
{
string key = null;
string val = "Message: TestMessage";
var result = producer.ProduceAsync(topicName, key, val).Result;
Console.WriteLine(val);
producer.Flush(100);
}
}
}
}
@johnchristudass - Is this resolved ? If yes, can you please share the fix. I am trying something similar.
@petersonnek - Do we need specify KeyTab in .NET Core app running on Windows and Kafka Broker on Linux?
@mhowlett - Do we have example of SASL-SSL configuration at .net client? Can you please add details , if available.
If you're running on linux, you will need a keytab
@etzellava Did you ever get around this issue? I'm seeing the same error logs.
@etzellava Did you ever get around this issue? I'm seeing the same error logs.
No, i tried many times but in the end i used webservice made in java
instead of using kerberos is there any other way to authenticate .net client consumer running in windows to connect to Kafka server in Linux environment ?
like AD based authentication or simple user id and password authentication ?
For ID/Passoword, your Kafka cluster would have to be set up with SASL_SCRAM or SASL_PLAIN. However, neither of those would integrate with AD directly. You'd need something like the LDAP Authorizer from Confluent to sync in IDs.
@petersonnek can we expect anytime soon the update of Confluent.Kafka to support Kerberos authentication (using keytabs) ?
I think this issue should be re-opened until the examples on this page: https://github.com/confluentinc/confluent-kafka-dotnet/issues/594 are completed and we have good working example with SASL+SSL Kerberos, which is what mostly everyone is asking about here. With keytabs or without.
Coukd you folks guide me to a reouce ? is this issue fixed ?
I'm also using windowsachin to connect to kafka with keytabs,
Here's a function we've been using that works for both Windows (integrated) and Linux (keytabs)
private ConsumerConfig MakeConsumerConfig()
{
var settings = new ConsumerConfig() {
BootstrapServers = Configuration["Kafka:BrokerList"],
GroupId = Configuration["Kafka:ConsumerGroup"],
ClientId = System.Environment.MachineName,
//Debug = "consumer,cgrp,topic,fetch", //detailed consumer logging: "consumer,cgrp,topic,fetch"
EnableAutoOffsetStore = false,
EnableAutoCommit = true,
AutoOffsetReset = AutoOffsetReset.Earliest, // if offset is invalid, start over
SecurityProtocol = SecurityProtocol.SaslSsl,
SaslMechanism = SaslMechanism.Gssapi,
SaslKerberosServiceName = "Kafka"
settings.SslCaLocation = "ca-bundle.pem";
};
if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
settings.SaslKerberosKeytab = "/etc/secrets/ID.keytab";
settings.SaslKerberosPrincipal = Configuration["KerberosPrincipal"];
}
return settings;
}
hi - can you share config for windows .net producer using kerberos and keytab. thank you
Most helpful comment
No, i tried many times but in the end i used webservice made in java