Jaeger: Connect/Integrate jaeger-ingest and jaeger-collector with a Kafka-Enabled Azure Event Hub

[awesomeinsight]

Requirement - what kind of business use case are you trying to solve?

Connect/Integrate jaeger-ingest and jaeger-collector with a Kafka-Enabled Azure Event Hub which is provided by a PaaS. However, the kafka service provided by the platform only supports SASL authentication with e.g. the following settings:

sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username="$ConnectionString"
password="{MY.EVENTHUBS.CONNECTION.STRING}";

see: https://github.com/Azure/azure-event-hubs-for-kafka

Lots of other tools like fluent-bit, filebeat, logstash or flink, spark,... already support to connect to an Kafka enabled Azure Event Hub.

Problem - what in Jaeger blocks you from solving the requirement?

Jaeger does not support to connect to an Azure Event Hub that is Kafka-enabled currently.
So its not possible to realize an architecture like the one described here: https://www.jaegertracing.io/img/architecture-v2.png.

Proposal - what do you suggest to solve the problem or improve the existing situation?

There is already a PR (https://github.com/jaegertracing/jaeger/issues/1966) which is adding Basic Auth for Kafka to jeager-collector. This could be a starting point / motivation to support all other neccesary settings/properties mentioned avove to connect to a Kafka enabled Azure Event Hub too.

[chandresh-pancholi]

I am picking it up next.

[awesomeinsight]

Sounds great! If you have any questions regarding the kafka enabled Azure Event Hub please don't hestitate to ask - also if I can help/assist you in any other direction.

[Quang-OLLI]

"Unable to create consumer","error":"invalid versionSASL_SSL" I got this error while connecting to kafka. Any idea how to fix this one ?

[narsii4u]

Any update on this? we also have the same requirement. Thanks.

[Quang-OLLI]

I had to override the report spans function to get it send messages to Kafka instead of jaeger-agent

[vamshi67]

Sounds great! If you have any questions regarding the kafka enabled Azure Event Hub please don't hestitate to ask - also if I can help/assist you in any other direction.

I need to use confluent Kafka (Provisioned on Azure) as storage type. How can I pass sasl info from jaeger-collector?

[jpkrohling]

Not sure if there's anything special for Confluent Kafka, but you can specify the username/password via flags. All possible flags can be seen via docker run -e SPAN_STORAGE_TYPE=kafka jaegertracing/jaeger-collector:1.18 --help:

      --kafka.producer.plaintext.password string       The plaintext Password for SASL/PLAIN authentication
      --kafka.producer.plaintext.username string       The plaintext Username for SASL/PLAIN authentication