Strimzi-kafka-operator: Kafka Connect Resource vs. Kafka Resource

[Yepez1997]

As I am reading through the documentation I am trying to figure the difference between a cluster as kind: Kafka Connect Resource versus Kafka Resource ? I have an existing Kafka Resource running, and I want to install a connector to run elastic search aside. Of course I have to deploy a Kafka Connector resource. Do I start from scratch and create a Kafka Connect Deployment instead of a Kafka Deployment. I have TLS + AUTH in my current Kafka Deployment, do the same certs apply to Kafka Connect Deployment ?

[Yepez1997]

The documentation states: _A Kafka Connect cluster always works in combination with a Kafka cluster. A Kafka cluster is specified as a list of bootstrap servers. On Kubernetes, the list must ideally contain the Kafka cluster bootstrap service named cluster-name-kafka-bootstrap, and a port of 9092 for plain traffic or 9093 for encrypted traffic._ Assuming Kafka Connect creates a bootstrap server also ?

[scholzj]

• The Kafka resource is used to manage Kafka clusters (as in Kafka brokers + Zookeeper).
• The KafkaConnect resource is used to deploy the KafkaConnect cluster. You will always need to configure an address of a Kafka cluster where it connects. But while we will be of course happy if you use Strimzi to run the Kafka cluster, you can also connect the KafkaConnect resource to other Kafka clusters. There are users using it with Amazon MSK, we even got it working with Azure EventHubs.
• The KafkaConnector is used to create an instance of a connector inside KafkaConnect

So basically, in most cases you:
1) Deploy the Kafka cluster using the Kafka CR (or deploy Kafka in some other way)
2) Deploy the KafkaConnect cluster using KafkaConnect CR and configure it to connect to the Kafka cluster from step 1
3) Create the KafkaConnector and have it deployed on the Connect cluster from step 2

[Yepez1997]

Thank you for the quick response, so in other words KafkaConnect Resource cluster does not create an addition ZooKeeper and Kafka ? Also to follow up, will the same certs & tls I have set up for my Kafka Resource apply to the Kafka Connect Resource as well ? I was just confused as to why both KafkaConnect Resource and Kafka Resource are both considered clusters. As long as this portion is configured: spec.bootstrapServers: < bootstrap-servers-kafka-resource > inside the Connect Resource, it is fine to use both "clusters" alongside (as mentioned in step 2)

[scholzj]

so in other words KafkaConnect Resource cluster does not create an addition ZooKeeper and Kafka ?

Correct, it doesn't. Just Connect.

Also to follow up, will the same certs & tls I have set up for my Kafka Resource apply to the Kafka Connect Resource as well ? I was just confused as to why both KafkaConnect Resource and Kafka Resource are both considered clusters. As long as this portion is configured: spec.bootstrapServers: < bootstrap-servers-kafka-resource > inside the Connect Resource, it is fine to use both "clusters" alongside ?

It really depends what did you configured in the Kafka CR. If you share your Kafka CR and KafkaConnect CR I might be able to give more specific help. But basically if you care only about encryption, then your Kafka CR might look something like this: https://github.com/strimzi/strimzi-kafka-operator/blob/master/examples/kafka/kafka-persistent.yaml where the important part is:

    listeners:
      tls: {}

And the matching KafkaConnect resource is here https://github.com/strimzi/strimzi-kafka-operator/blob/master/examples/connect/kafka-connect.yaml with the important part being:

  bootstrapServers: my-cluster-kafka-bootstrap:9093
  tls:
    trustedCertificates:
      - secretName: my-cluster-cluster-ca-cert
        certificate: ca.crt

Which uses the TLS port 9093 and the secret with the TLS certificate.

If you want also authentication, then you can have a look at these examples https://github.com/strimzi/strimzi-kafka-operator/tree/master/examples/security which again have both matching Kafka and KafkaConnect CRs.

[Yepez1997]

All makes sense now, it is straight forward from here, I've looked at plenty of examples for auth and tls with Kafka Connect.

kafka.yaml

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-kafka-cluster
spec:
  kafka:
    version: 2.4.0
    replicas: 1
    listeners:
      plain: {}
      tls: {}
      external:
        type: loadbalancer
        tls: true
        # authentication:
        #   type: tls
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.4"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    userOperator: {}
    topicOperator: {}

kafka-connect.yaml

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
  name: kafka-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 2.4.0
  replicas: 1
  bootstrapServers: my-kafka-cluster-kafka-bootstrap:9093
  config:
    group.id: connect-cluster
    offset.storage.topic: strimzi-connect-cluster-offsets
    config.storage.topic: strimzi-connect-cluster-configs
    status.storage.topic: strimzi-connect-cluster-status
  tls:
    trustedCertificates:
      - secretName: kafka-tls-client-credentials # from kafka.yaml
        certificate: ca.crt
  logging:
    type: external
    name: connect-logging-configmap

One last question, will there ever be support for a strimizi supported connector for elastic search ? I'm following this as an example for elastic search (of course, i'd download the plugins): https://strimzi.io/blog/2020/01/27/deploying-debezium-with-kafkaconnector-resource/

[scholzj]

The configs look good 👍

One last question, will there ever be support for a strimizi supported connector for elastic search ? I'm following this as an example for elastic search (of course, i'd download the plugins): https://strimzi.io/blog/2020/01/27/deploying-debezium-with-kafkaconnector-resource/

You should be able to use Strimzi with any connectors which would work with Apache Kafka. You always have to add them to the container image but once you add them they should work. I know some people used the Camel ElasticSearch connector (https://camel.apache.org/camel-kafka-connector/latest/connectors/camel-elasticsearch-rest-kafka-sink-connector.html) with Strimzi. And I think some people also used the Confluent one.

We do not really plan to bundle any connectors directly into the images since that would make the container images too big as you would have always some connectors which you will not use.