Strimzi-kafka-operator: [Kafka - SSL handshake failed error on 9093 port from producer shell script call within kafka broker POD] ...

[vperi1730]

Hi Team,

I am testing a use case of authentication using TLS port 9093 with all the required certificates. However I am receiving SSL handshake, Following are the steps which I followed, need help technically to identify the issue behind.

`1) kubectl get secrets -n kafka-operator1 my-cluster-cluster-ca-cert -o jsonpath='{.data.ca.crt}' | base64 -id > ca.crt

2) kubectl get secrets -n kafka-operator1 my-cluster-cluster-ca -o jsonpath='{.data.ca.key}' | base64 -id > ca.key

3) kubectl get secrets -n kafka-operator1 my-cluster-kafka-brokers -o jsonpath='{.data.my-cluster-kafka-0.crt}' | base64 -id > ca_k0.crt

4) kubectl get secrets -n kafka-operator1 my-cluster-kafka-brokers -o jsonpath='{.data.my-cluster-kafka-1.crt}' | base64 -id > ca_k1.crt

5) kubectl get secrets -n kafka-operator1 my-cluster-kafka-brokers -o jsonpath='{.data.my-cluster-kafka-2.crt}' | base64 -id > ca_k2.crt

6) keytool -keystore client.truststore.p12 -storepass 123456 -noprompt -alias my-cluster-kafka-0 -import -file ca_k0.crt

7) keytool
-import
-file ca_k1.crt
-keystore client.truststore.p12
-alias my-cluster-kafka-1
-storepass 123456
-noprompt

8) keytool
-import
-file ca_k2.crt
-keystore client.truststore.p12
-alias my-cluster-kafka-2
-storepass 123456
-noprompt

9) keytool
-import
-file ca.crt
-keystore client.truststore.p12
-alias ca
-storepass 123456
-noprompt
`

All the above commands are doing 1 thing finally, creating client.truststore.p12 which i am placing inside /tmp/ folder and calling the producer.sh as below.

[kafka@my-cluster-kafka-0 kafka]$ ./bin/kafka-console-producer.sh --broker-list my-cluster-kafka-bootstrap.kafka-operator1.svc.cluster.local:9093 --topic happy-topic

--producer-property security.protocol=SSL
--producer-property ssl.truststore.type=PKCS12
--producer-property ssl.truststore.password=123456
--producer-property ssl.truststore.location=/tmp/prod/client.truststore.p12
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThre
ads=N
[2020-05-15 16:23:36,698] ERROR [Producer clientId=console-producer] Connection to node -1 (my-cluster-kafka-bootstrap.kafka-operator1.svc.cluster.local/10.12.4.238:9093) failed authentication
due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2020-05-15 16:23:36,996] ERROR [Producer clientId=console-producer] Connection to node -1 (my-cluster-kafka-bootstrap.kafka-operator1.svc.cluster.local/10.12.4.238:9093) failed authentication d
ue to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2020-05-15 16:23:37,626] ERROR [Producer clientId=console-producer] Connection to node -1 (my-cluster-kafka-bootstrap.kafka-operator1.svc.cluster.local/10.12.4.238:9093) failed authentication d
ue to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

[scholzj]

First of all, can you share the Kafka custom resource? You talk about authentication - does that mean client authentication? Or only sever authentication? You specify only the truststore, so that will only verify the servers identity and not do any TLS client authentication.

As for the steps:

• I would recommend to run the client in a separate pod
• For the truststore, you should need only the steps 1 + 9. The truststore needs to have only the public key of the Strimzi CA which is used to sign the broker certificates. It does not need the private key or the node certificates.

For the TLS client authentication, you need to crete a user. Assuming you have the User OPerator enabled, you can do that by creating a KafkaUser custom resource like this:

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
  name: my-user
  labels:
    strimzi.io/cluster: my-cluster
spec:
  authentication:
    type: tls

Once the user is created, the user operator will create a secret with the same name. From it, you can extract the public and private keys of the user or directl get the P12 file if you use recent version of Strimzi. And you have to use that as the ssl.keystore in the Kafka client.

[vperi1730]

Kafka Custom Resource.

Kafka/Zookeeper CR

kafkaYaml: |-
apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: my-cluster spec: kafka: replicas: 3 listeners: plain: {} tls: authentication: type: tls external: type: loadbalancer tls: true authentication: type: tls storage: type: persistent-claim size: 10Gi deleteClaim: false config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 zookeeper: replicas: 3 storage: type: persistent-claim size: 10Gi deleteClaim: false entityOperator: topicOperator: {} userOperator: {}

1) We wanted to cover initially server auth and then the client auth. Yes, we do have the custom resources for creating kafkauser and kafkatopic.

2) For 2 way authentication (let's say to produce and consume a topic) what all cert's are required w.r.t trust and keystore if you can put some thoughts there.

[scholzj]

This part enabled the TLs client authentication:

          authentication:
            type: tls

So if you want to start with server auth only (which means basically regular TLS encryption), you have to remove it and do the steps 1 and 9 from your original post. The client should be configured well for that.

For the client auth as I said in my first answer ... you will need to create the user and the keystore as I described there.

[vperi1730]

Thanks for the information...this helps.

Another query here is when you say the client should be configured well, can't we test using the producer shell script to verify the regular TLS auth (or) is it that you are recommending something as shown below....

kind: Pod
apiVersion: {API_Version}
metadata:
  name: client-pod
spec:
  containers:
  - name: client-name
    image: client-name
    volumeMounts:
    - name: secret-volume
      mountPath: /data/p12
    env:
    - name: SECRET_PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: my-password
  volumes:
  - name: secret-volume
    secret:
      secretName: my-cluster-cluster-cert

if YES does the env portion above represents the cluster CA cert along with its crt password which is populated in the secret created?

[scholzj]

You can use the regular consumers of course. But running it from a separate pod is better. Starting another JVM inside the pod can cause resource issues to the brokers for example. So especially in production, you should avoid it whenever possible.

[vperi1730]

1-way auth has worked successfully. thanks for the help.

We started testing 9094 by creating a topic and a user. However, we ended up in an SSL handshake. The following are the steps we performed, need help where we went wrong.

happy-user is the kafkauser
happy-topic is the kafkatopic.

1) kubectl get secrets -n kafka-operator1 happy-user -o jsonpath='{.data.user.crt}' | base64 -id > user.crt

2) kubectl get secrets -n kafka-operator1 happy-user -o jsonpath='{.data.user.key}' | base64 -id > user.key

3) openssl pkcs12 -export -in user.crt -inkey user.key -name user.p12 -password pass:123456 -out producer.p12

4) keytool -importkeystore -alias user.p12 -deststorepass 123456 -destkeystore producer.keystore.p12 -srcstorepass 123456 -srckeystore producer.p12 -srcstoretype PKCS12 -deststoretype PKCS12

Test command:

kafka@my-cluster-kafka-0 kafka]$ ./bin/kafka-console-producer.sh --broker-list my-cluster-kafka-external-bootstrap.kafka-operator1.svc.cluster.local:9094 --topic happy-topic

--producer-property security.protocol=SSL
--producer-property ssl.truststore.type=PKCS12
--producer-property ssl.keystore.type=PKCS12
--producer-property ssl.truststore.password=123456
--producer-property ssl.keystore.password=123456
--producer-property ssl.truststore.location=/tmp/prod/client.truststore.p12
--producer-property ssl.keystore.location=/tmp/prod/producer.keystore.p12
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThre
ads=N
[2020-05-16 12:38:11,011] ERROR [Producer clientId=console-producer] Connection to node -1 (my-cluster-kafka-external-bootstrap.kafka-operator1.svc.cluster.local/10.12.1.33:9094) failed authent
ication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2020-05-16 12:38:11,155] ERROR [Producer clientId=console-producer] Connection to node -1 (my-cluster-kafka-external-bootstrap.kafka-operator1.svc.cluster.local/10.12.1.33:9094) failed authenti
cation due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2020-05-16 12:38:11,532] ERROR [Producer clientId=console-producer] Connection to node -1 (my-cluster-kafka-external-bootstrap.kafka-operator1.svc.cluster.local/10.12.1.33:9094) failed authenti
cation due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2020-05-16 12:38:12,427] ERROR [Producer clientId=console-producer] Connection to node -1 (my-cluster-kafka-external-bootstrap.kafka-operator1.svc.cluster.local/10.12.1.33:9094) failed authenti
cation due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

[scholzj]

If you pass the client the Java system property -Djavax.net.debug=ssl it will show the details of the TLS handshake and why it fails. That should tell us more.

_Note: Fixed to ssl_

[vperi1730]

I am a little unsure of how do we pass this. I tried something like below, didn't work.

java -Djavax.net.debug=tls SSLSocketClientWithClientAuth

or tried like this

--producer-property -Djavax.net.debug=tls

Could you please suggest.

[scholzj]

I think that if you are using the console producer / consumer, you can do:

export KAFKA_OPTS="-Djavax.net.debug=ssl"
./bin/kafka-console-producer.sh ...

[vperi1730]

Thanks for the information, it helped and worked with detailed logs in the console. A couple of next questions I have is...

Q1) In the logs, I have seen the exception - java.security.cert.CertificateException: No subject alternative DNS name matching my-cluster-Kafka-external-bootstrap.kafka-operator1.svc.cluster.local found. In the logs under the Subject alternative DNS list, i didn't find it. In my Kafka pod list, I can see the below info.

my-cluster-Kafka-external-bootstrap 34.75.42.230(External IP) : 9094 TCP.

Wanted to know why is that?

SubjectAlternativeName [
DNSName: my-cluster-kafka-brokers.kafka-operator1.svc
DNSName: my-cluster-kafka-brokers.kafka-operator1.svc.cluster.local
DNSName: my-cluster-kafka-brokers
DNSName: my-cluster-kafka-brokers.kafka-operator1
DNSName: my-cluster-kafka-1.my-cluster-kafka-brokers.kafka-operator1.svc.cluster.local
IPAddress: 34.75.42.230
IPAddress: 34.75.139.132
DNSName: my-cluster-kafka-bootstrap.kafka-operator1.svc
DNSName: my-cluster-kafka-bootstrap.kafka-operator1.svc.cluster.local
DNSName: my-cluster-kafka-bootstrap
DNSName: my-cluster-kafka-bootstrap.kafka-operator1
DNSName: my-cluster-kafka-1.my-cluster-kafka-brokers.kafka-operator1.svc ]

2) I am able to successfully produce and consume using the external bootstrap IP address, it worked.

Q3) How do we check if the user who has produced and consumed is the same kafkauser we have created???, Do we need to enable some logging or how is it??

[scholzj]

Q1:

You have 3 listeners configured in your broker:

     listeners:
        plain: {}
        tls:
          authentication:
            type: tls
        external:
          type: loadbalancer
          tls:  true
          authentication: 
            type: tls          

The plain and tls are supposed to be used from inside the Kubernetes cluster and you can use the bootstrap URL my-cluster-kafka-bootstrap.kafka-operator1.svc:9093 (for tls) or my-cluster-kafka-bootstrap.kafka-operator1.svc:9092 (for plain - without TLS encryption). So when you connect from inside Kubernetes, you should use one of these.

The external listener is designed to be used from outside of the Kubernetes cluster. And to connect, you have to (in your case since you use type: loadbalancer) use the load balancer address (in your case IP, in other cases it might be DNS name).

So you should pick the right address depending where the client is running. You can use the external listener also from inside, but:

• You should use the loadbalancer address
• It will be probably slower and sometimes more expensive because the traffic will flow through the loadbalancers. So in general you should always prefer to the internal tls interface instead. The SAN list in the certificates also corresponds to this.

Q3

I'm sure the logs on the brokers or clients will show the username somwhere. But to be honest, I'm not sure where to look out of my head. You might also need to increase the log level (https://strimzi.io/docs/latest/full.html#con-kafka-logging-deployment-configuration-kafka) - I do not think it is printed by default.

TBH, I don't know. If you connect

[vperi1730]

Q3 has worked, thanks for the information.

For Q1 I have tried in the following way.

1) I have created a private DNS in the google cloud which is bind to the load balancer IP.
2) I was able to dig the DNS name and could see the following mapping.

dig kafka.viswa.com
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> kafka.viswa.com
;; QUESTION SECTION:
;kafka.viswa.com. IN A
;; ANSWER SECTION:
kafka.viswa.com. 60 IN A 35.243.152.49

3) We have updated the Kafka custom resource with the following change.

overrides: bootstrap: dnsAnnotations: external-dns.alpha.kubernetes.io/hostname: kafka.viswa.com external-dns.alpha.kubernetes.io/ttl: "60"

4) We have re-deployed the operator after all these changes, however, when we are hitting the producer shell script using the DNS name it is throwing back SSL handshake failed.

./bin/kafka-console-producer.sh --broker-list kafka.viswa.com:9094 --topic happy-topic
--producer-property security.protocol=SSL
--producer-property ssl.truststore.type=PKCS12
--producer-property ssl.keystore.type=PKCS12
--producer-property ssl.truststore.password=123456
--producer-property ssl.keystore.password=123456
--producer-property ssl.truststore.location=/tmp/prod/client.truststore.p12
--producer-property ssl.keystore.location=/tmp/prod/producer.keystore.p12

Below is the exception noticed...java.security.cert.CertificateException: No subject alternative DNS name matching kafka.viswa.com found.

I need your help...

[scholzj]

The external DNS annotations are for the ExternalDNS tool. They are not used by Strimzi - we just pass them to the serviced where the ExternalDNS utility can pick them up. So if you don't use External DNS you do not need them really.

If you want the bootstrap address to be added to the certificate you should do aomething like this:

listeners:
  #...
  external:
    type: route
    authentication:
      type: tls
    overrides:
      bootstrap:
        address: kafka.viswa.com

See https://strimzi.io/docs/latest/full.html#con-kafka-broker-external-listeners-addresses-deployment-configuration-kafka for more details.

[vperi1730]

Fantastic, thank you so much for taking me in the right direction, it has worked :). I will be verifying all the use cases and will revert back for any other help.

Thanks again.....

[vperi1730]

Need other help related to the consumer group.

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
  name: happy-user-consumer
  labels:
    strimzi.io/cluster: cluster1
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: happy-topic
        operation: Create
      - resource:
          type: topic
          name: happy-topic
        operation: Describe
      - resource:
          type: group
          name: happy-user-consumer-group
        operation: Read

I am running the following script with group created during kafkauser creation....
[kafka@my-cluster-kafka-0 kafka]$ ./bin/kafka-console-consumer.sh --bootstrap-server kafka.viswa.com:9094 --topic happy-topic --group happy-user-consumer-group

--consumer-property security.protocol=SSL
--consumer-property ssl.truststore.type=PKCS12
--consumer-property ssl.keystore.type=PKCS12
--consumer-property ssl.truststore.password=123456
--consumer-property ssl.keystore.password=123456
--consumer-property ssl.truststore.location=/tmp/prod/client.truststore.p12
--consumer-property ssl.keystore.location=/tmp/prod/consumer.keystore.p12 --from-beginning

However, the messages which I am producing are not reaching to the consumer in spite of all the ACL's activated and certificates are in place.

Q) Do we need to add the topic to the consumer group so that it consumes the messages, If YES how do we do that?

[kafka@my-cluster-kafka-0 kafka]$ ./bin/Kafka-ACLS.sh --authorizer-properties zookeeper.connect=127.0.0.1:2181 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX: ParallelGCThreads=N
Current ACLs for resource Group:LITERAL:happy-user-consumer-group:
User: CN=happy-user-consumer has Allow permission for operations: Read from hosts: *
Current ACLs for resource Topic:LITERAL:happy-topic:
User: CN=happy-user has Allow permission for operations: Create from hosts: *
User: CN=happy-user has Allow permission for operations: Describe from hosts: *
User: CN=happy-user-consumer has Allow permission for operations: Read from hosts: *
User: CN=happy-user has Allow permission for operations: Write from hosts: *
User: CN=happy-user-consumer has Allow permission for operations: Describe from hosts: *

[scholzj]

Please have a bit more look at the examples we provide which show the rights needed for consuming and producing: https://github.com/strimzi/strimzi-kafka-operator/blob/master/examples/user/kafka-user.yaml ... it looks like your consumer has no right to read from the topic.

[vperi1730]

We have tried with the same approach, Groupauthorizationexception is observed. Let us know if we are missing something. Need help.

1) Kafkauser Custom Resource.

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
  name: happy-user-consumer
  labels:
    strimzi.io/cluster: cluster1
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: happy-topic
          patternType: literal
        operation: Read
        host: "*"
      - resource:
          type: topic
          name: happy-topic
          patternType: literal
        operation: Describe
        host: "*"
     - resource:
          type: group
          name: happy-user-consumer-group
          patternType: literal
        operation: Read
        host: "*"

2) Consumer call with all the parameters and group name passed----
./bin/kafka-console-consumer.sh --bootstrap-server kafka.viswa.com:9094 --topic happy-topic --group happy-user-consumer-group

Error: org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: console-consumer-4369

I am not sure where is it pointing to console-consumer group. Also with the below script, I am not able to list the consumer groups. Do we have any other way to see the groups?

./bin/kafka-consumer-groups.sh --bootstrap-server kafka.viswa.com:9094 --command-config /tmp/prod/consumer.properties --list

[scholzj]

It looks like it ignores the --group option. But no idea why. You will probably need to debug that locally. I didn't had the problem when I was trying it last time.

[vperi1730]

Ok, basically I am able to produce the message on the Kafka topic successfully, however, on the consumer end, I am not able to view them because of Groupauthexception. This is observed even when I am not passing any group name to the shell script, something like below.

./bin/kafka-console-consumer.sh --bootstrap-server kafka.viswa.com:9094 --topic happy-topic --
--command-config /tmp/prod/consumer.properties --list

In the logs, I have noticed it is considering a default consumer group that doesn't have access to my topic is what I assume.

Looking for a workaround.

[vperi1730]

This is resolved, thanks for all the support, helped to gain lot of practical knowledge.

[scholzj]

So ... what was the problem with the consumer group? Did you figured out why was it ignoring the --group option? In case someone else has the same problem.

[vperi1730]

Ya, we have figured it out. The important point here to remember is we can't create a consumer group using the shell script, so when we are trying to consume the topic using console-consumer.sh, it was defaulting to an existing consumer group which our topic and user don't have access too.

Hence we need to do something like this.

1)
./bin/kafka-console-consumer.sh --bootstrap-server kafka.dns.acl.com:9094 --topic acl-topic
--consumer-property security.protocol=SSL
--consumer-property ssl.truststore.type=PKCS12
--consumer-property ssl.keystore.type=PKCS12
--consumer-property ssl.truststore.password=123456
--consumer-property ssl.keystore.password=123456
--consumer-property group.id=acl-user-consumer-grp
--consumer-property ssl.truststore.location=/tmp/prod/client.truststore.p12
--consumer-property ssl.keystore.location=/tmp/prod/consumer.keystore.p12 --from-beginning

Property :::: --consumer-property group.id=acl-user-consumer-grp -> This will create the consumer group and also it has to be added in the kafkauser CR under resource:type->group as you mentioned.

2) ./bin/kafka-consumer-groups.sh --bootstrap-server kafka.dns.acl.com:9094 --command-config /tmp/prod/consumer.properties --describe --all-groups - This will show the list of consumer groups we have created and the topics connected to it.

Hope this helps.

[scholzj]

That helps. Thanks. I wonder if the --group is some bug in Kafka ...what version did you tried it with? Because it definitely used to work for me in the past.

[vperi1730]

--group I haven't tried today as the consumer-property itself has solved my use case.

I have used the 2.4.0 Kafka version.

[scholzj]

Ok, great. Thanks.

[pascals-ager]

@vperi1730 It would be pretty awesome if you can make a PR to the examples section with the manifests for your set up.

[scholzj]

@pascals-ager What exactly are you looking for? I added some more security oriented examples the other day: https://github.com/strimzi/strimzi-kafka-operator/tree/master/examples/security