Strimzi-kafka-operator: Expose Kafka to the outside world

[scholzj]

It should be possible to expose Kafka to the outside of OpenShift / Kubernetes world.

---

Search term boost: broker external IP external-ip address addresses port ports ingress

[khannz]

@scholzj, hi there. Any ideas, when issue can be closed or maybe there is already known workaround for exposing kafka part of Strimzi?

[scholzj]

To be honest, this is currently not our highest priority at this point. We want to get first security working (encryption, authentication and authorization) and then we will probably get to this.

But PRs from community are of course always welcomed ;-).

[khannz]

d'oh... @scholzj deadlines kick hard so possibly PR wouldn't be mine ;]

In case I finally got what is wrong with kafka expose via PortNode in my env (also trying Yolean) - I think I'll figure out how it could be implemented in Strimzi.

[taylorsmithgg]

Any ETA on this?

[scholzj]

@taylorsmithgg I hope to get to it during September. (no promisses, just hopes ;-))

[taylorsmithgg]

@scholzj Sounds great. I'd love to see it working with Nginx Ingress Controller as well!

[scholzj]

TBH, I'm not sure what is the latest state of TCP / TLS passthrough in Nginx Ingress. If that is solved it should be possible to make it work even through Ingress. Although to be honest I expect the Ingress to take some price on the performance. So I guess ideally we should have a choice ... node ports for more performant setup which is a bit more complicated and Inrgess for _slower_ but easier to setup way.

[chrisjschultz]

@khannz - did you have any examples you can share for external cluster access? ta!

[chrisjschultz]

@scholzj - I am far from a K8n/OSE expert, but I am about to deploy this and plan to use External IP addresses on OSE rather that node ports (have not started the work yet). Do you think this is viable?

[scholzj]

Again, that would not really work out of the box. The Kafka clients need to be able to make a direct connections to the each specific broker. So you would need to expose every broker this way + configure it inside the brokers in the advertised.listeners field. But it is an interesting idea because I haven't thought about it yet.

[juniorgasparotto]

Any news on this topic? The closest I got was to update the service to create an external IP, but I don't know why, the configuration is lost seconds later.

oc patch svc my-cluster-kafka-bootstrap -p '{"spec":{"externalIPs":["192.168.0.10"]}}'
oc patch svc my-cluster-zookeeper-client -p '{"spec":{"externalIPs":["192.168.0.10"]}}'

[]s

[scholzj]

@juniorgasparotto On what topic? This issue is implemented for a long time. So it is not compltely clear what are you refering to.

[juniorgasparotto]

Sorry for not being clear, I am referring to the of being able to connect to kafka from outside the cluster. I already tried different ways and none of them worked and I believe that this issue is related to that, am I right?

Thanks

[scholzj]

So which exactly ways you tried? We have like 4 different ways all described in the docs: https://strimzi.io/docs/latest/full.html#assembly-kafka-broker-external-listeners-deployment-configuration-kafka ... as well as a blog post series: https://strimzi.io/2019/04/17/accessing-kafka-part-1.html

[juniorgasparotto]

Hello,

I did it, followed the document and it worked using a route.

However, I was unable to consume my topic in any way, would it be a collateral effect of this change?

1) It's work!

 ./kafka-console-producer.bat --broker-list my-cluster-kafka-bootstrap-project-kafka.192.168.0.10.nip.io:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.jks --topic brasil

2) This does not work, the prompt is stopped and does not display anything.

./kafka-console-consumer.bat --bootstrap-server my-cluster-kafka-bootstrap-project-kafka.192.168.0.10.nip.io:443 --topic teste

3) The next test stopped working after the modification, would this be a side effect?

oc -n project-kafka run kafka-producer -ti --image=strimzi/kafka:0.16.2-kafka-2.4.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --broker -list my-cluster-kafka-bootstrap:9091 --topic teste

ERROR: org.apache.kafka.common.errors.TimeoutException: Topic teste not present in metadata after 60000 ms.

Thanks

[scholzj]

In the first command, you configured the TLS. However in the second you don't. That is why the producer works and the consumer doesn't.

In the third command, you are not supposed to connect to port number 9091. That port is used only for replication and cluster synchronization. If you have something like this in your Kafka CR:

listeners
  plain: {}
  external:
    # ...

you should be able to connect on port 9092.

[juniorgasparotto]

Now I understand!

1) Thanks a lot for the help, I passed the certificate on the consumer and it worked.

2) I ended up removing the "listeners -> plain = {}" configuration and caused the problem of not being able to use the producer and consumer within the cluster via service.

Thank you for your help