Strimzi-kafka-operator: Deployed through repo, Connection Reset by peer when connecting to ZK

[eosantigen]

Describe the bug
I followed the steps below and deployment.apps/strimzi-topic-operator gets stuck at establishing a connection to ZK .

The error;

2020-04-01 08:18:40 INFO  ZooKeeper:442 - Initiating client connection, connectString=kafka-operator-zookeeper-client:2181 sessionTimeout=20000 watcher=org.I0Itec.zkclient.ZkClient@2ce2eb34
2020-04-01 08:18:40 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
2020-04-01 08:18:41 INFO  ClientCnxn:1025 - Opening socket connection to server kafka-operator-zookeeper-client/10.0.87.243:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-01 08:18:41 INFO  ClientCnxn:879 - Socket connection established to kafka-operator-zookeeper-client/10.0.87.243:2181, initiating session
2020-04-01 08:18:41 WARN  ClientCnxn:1164 - Session 0x0 for server kafka-operator-zookeeper-client/10.0.87.243:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_222]
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_222]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_222]
    at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_222]
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_222]

To Reproduce

1. Downloaded version (currently) 0.14.0 from official Strimzi Github repository through Releases. (a zip).
2. sed -i 's/namespace: .*/namespace: kafka-operator/' install/cluster-operator/*RoleBinding*.yaml
3. kubectl apply -f install/cluster-operator/
4. Wait for the resources to come up and do a kubectl get all -n kafka-operator
5. Check the service name and add it to the respective fields in the deployment file of topic-operator folder.
6. `kubectl apply -f install/topic-operator

Expected behavior
Connection from deployment.apps/strimzi-topic-operator to ZK successful and pods all running with 1/1

Environment (please complete the following information):

• Strimzi version: 0.14.0 _(do not wish to upgrade at the moment)_
• Installation method: original YAML files from the Releases repo official
• Kubernetes cluster: Kubernetes 1.14
• Infrastructure: AKS

YAML files and logs
The following snippet is the connection details to the K8s svc's.

          env:
            - name: STRIMZI_RESOURCE_LABELS
              value: "strimzi.io/cluster=kafka-operator"
            - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS
              value: kafka-operator-kafka-bootstrap:9092
            - name: STRIMZI_ZOOKEEPER_CONNECT
              value: kafka-operator-zookeeper-client:2181

Have I missed something? Many Thanks!

[scholzj]

I'm not sure I fully understand the steps to reproduce and what exactly are you trying to do.

Are you trying to deploy the TOpic Operator on its own and use it with your own Kafka cluster? In that case you need to use the files from install/topic-operator and configure it to connect to your Zookeeper and Kafka.

Or are you trying to use it with Kafka cluster deployed by Strimzi? In that case you do not need to use the files from install/topic-operator at all. The Topic Operator is deployed as part of your Kafka cluster as described here: https://strimzi.io/docs/latest/full.html#deploying-the-topic-operator-using-the-cluster-operator-str

[eosantigen]

Thank you for replying. Yes, I have deployed the files from install/topic-operator and configured it to connect to my kafka-Zk cluster. (which also has been deployed by the Strimzi official manifests.) . You are write, I don't need to deploy the topic-operator. Thanks on that. However, kafka cannot connect to ZK anyways. _Connection reset by peer._ I can't even connect through the ZK pod on localhost. [kafka@kafka-operator-zookeeper-0 kafka]$ bin/kafka-topics.sh --list --zookeeper localhost:2181

What am I missing..?

[scholzj]

So as for the topic operator, to use it with Strimzi cluster you have to deploy it through the Kafka custom resource. The standalone install from install/topic-operator is for using it with your own Kafka cluster. To use that with Stirmzi cluster you would need to deal witht he security setup etc. So just deploy it from the Kafka custom resource and it should be deployed and configured for you (it will be part of the entity-operator pod).

[kafka@kafka-operator-zookeeper-0 kafka]$ bin/kafka-topics.sh --list --zookeeper localhost:2181

The Zookeeper is secured using an TLS proxy. So you cannot access it over localhost:2181 from the Zookeeper pods. You can:

• Exec into one of the Kafka pods and do bin/kafka-topics.sh --list --zookeeper localhost:2181
• Exec into one of the Zookeeper pods and do bin/kafka-topics.sh --list --zookeeper localhost:2181X where the X in the port number will be the index of the pod. So e.g. in kafka-operator-zookeeper-0 it would be bin/kafka-topics.sh --list --zookeeper localhost:21810