Cmak: Support Kafka 0.10+

[mcandre]

• [ ] Test basic features against a Kafka v0.10.1.0 cluster
• [ ] Accept a list of broker nodes, and read data from them rather than zk nodes

[Ashish123gs]

How do we manage permissions ?
Can KM display topic offset and consumer lags for a secured SASL kerberos cluster ?

[djenriquez]

Would definitely love this also.

By the way...is this project still supported? 5 months of inactivity seems a little long...

[jomach]

+1 on this

[hakamairi]

Is this project alive?

[krishna81m]

Any update?

[mslee89]

When will this project be updated?

[patelh]

It already has kerberos support. Just need to provide the config.

[Ashish123gs]

Thanks.

Is there any example of the config file ?

[kandeshvari]

I'm trying to configure kafka-manager with SASL_PLAINTEXT kafka

# cat jaas.conf 
KafkaClient {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin";
};

# cat consumer.properties 
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

run as

bin/kafka-manager -Dhttp.port=80 -Djava.security.auth.login.config=/etc/kafka-manager/jaas.conf

and gain infinite

[warn] o.a.k.c.NetworkClient - Bootstrap broker kafka-es2:9092 disconnected
[warn] o.a.k.c.NetworkClient - Bootstrap broker kafka-es1:9092 disconnected
[warn] o.a.k.c.NetworkClient - Bootstrap broker kafka-es3:9092 disconnected
[warn] o.a.k.c.NetworkClient - Bootstrap broker kafka-es2:9092 disconnected
...

1. Can someone provide me with a working config?
2. All stats gathers by JMX. Why kafka-manager tries to connect to kafka cluster directly?

[stevenschlansker]

Please let's not mix in random configuration issues into this issue. "How do I configure feature X" is really a question for the mailing list, not piling on an unrelated bug. Thanks!

[cl0udgeek]

Any update on when the SSL_SASL feature will be available?

[igalvit]

@stevenschlansker Is there a mailing list? Do you know where I can find it?

[igalvit]

Is there a way to config SSL_SASL for the brokers and not only for the zookeeper?

[basvdl]

@kandeshvari did you got it working? I'm stuck with the same...

[kandeshvari]

@basvdl yes, I've fix it after some shamanic actions but.. unfortunately I totally forgot this magic :(
I look in my production configs - they are the same as in https://github.com/yahoo/kafka-manager/issues/344#issuecomment-283363798

As I also remember I had an issue with broken zookeeper's data dir after adding SASL to kafka. After cleaning /var/lib/zookeeper/version-2 everything was fixed. Try this, maybe it will help.

[tanuj83]

@kandeshvari cleaning /var/lib/zookeeper/version-2 will cause issue to cleanup ZK data

[kandeshvari]

@tanuj83 exactly as I expect. I repeat - as I remember _MY_ problem was solved by cleaning ZK data. In my case nothing but runtime kafka config data stored in ZK. If you have any other data in _YOUR_ ZK - think twice before cleanup.

[rediceb]

config of jaas.conf

for zookeeper

Client {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafka"
password="kafka-sec";
};

for broker

KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="producer"
password="prod-sec";
};

consumer.properties

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

application.conf

kafka-manager.consumer.properties.file=/path/to/consumer.properties

kafka-manager-start.sh

exec ./bin/kafka-manager -Djava.security.auth.login.config=/path/to/jaas.conf -Dconfig.file=/path/to/application.conf "${OTHER_ARGS}" "${@}"

[basvdl]

I got rid of the infinite

[warn] o.a.k.c.NetworkClient - Bootstrap broker kafka-es2:9092 disconnected

But know i get the error below. I've a valid ticket (klist), and used the kafka-topics.sh to verify I can list all topics.

kafka_client_jaas.conf:

KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useTicketCache=true
   renewTicket=true
   serviceName="kafka";
};
Client {
   com.sun.security.auth.module.Krb5LoginModule required
   useTicketCache=true
   renewTicket=true
   serviceName="zookeeper";
};

Error:

[warn] k.m.a.c.KafkaManagedOffsetCache - Failed to process a message from offset topic on cluster test!
org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to configure SaslClientAuthenticator
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:106) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.network.Selector.connect(Selector.java:177) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:498) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:159) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:349) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:219) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:974) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
Caused by: org.apache.kafka.common.KafkaException: Failed to configure SaslClientAuthenticator
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.configure(SaslClientAuthenticator.java:124) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:102) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.network.Selector.connect(Selector.java:177) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:498) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:159) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:349) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:219) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
Caused by: org.apache.kafka.common.KafkaException: Failed to create SaslClient with mechanism PLAIN
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:139) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.configure(SaslClientAuthenticator.java:122) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:102) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.network.Selector.connect(Selector.java:177) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:498) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:159) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:349) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:219) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
Caused by: javax.security.sasl.SaslException: Cannot get userid/password
        at com.sun.security.sasl.ClientFactoryImpl.getUserInfo(ClientFactoryImpl.java:157) ~[na:1.8.0_131]
        at com.sun.security.sasl.ClientFactoryImpl.createSaslClient(ClientFactoryImpl.java:94) ~[na:1.8.0_131]
        at javax.security.sasl.Sasl.createSaslClient(Sasl.java:384) ~[na:1.8.0_131]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$1.run(SaslClientAuthenticator.java:135) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$1.run(SaslClientAuthenticator.java:130) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_131]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_131]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:130) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.configure(SaslClientAuthenticator.java:122) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:102) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
Caused by: javax.security.auth.callback.UnsupportedCallbackException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user.
        at org.apache.kafka.common.security.authenticator.SaslClientCallbackHandler.handle(SaslClientCallbackHandler.java:73) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at com.sun.security.sasl.ClientFactoryImpl.getUserInfo(ClientFactoryImpl.java:136) ~[na:1.8.0_131]
        at com.sun.security.sasl.ClientFactoryImpl.createSaslClient(ClientFactoryImpl.java:94) ~[na:1.8.0_131]
        at javax.security.sasl.Sasl.createSaslClient(Sasl.java:384) ~[na:1.8.0_131]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$1.run(SaslClientAuthenticator.java:135) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$1.run(SaslClientAuthenticator.java:130) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_131]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_131]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:130) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.configure(SaslClientAuthenticator.java:122) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]

[tanuj83]

Here is my solution if you have both PLAINTEXT and SSL/SASL, it works for me when I change order
#310