Fluent-bit: Kafka authentication and TLS support in fluent-bit kafka output plugin.

[abhishekkumarsingh1]

It seems as of now there is no support of TLS and authentication for kafka_out plugin. This support is important for forwarding logs securely to kafka.

[mplachter]

Is there any work to include SSL through librdkafka properties?

[gabe-sorensen]

This should be supported. I was able to get it to connect with the following section of properties:

rdkafka.security.protocol    ssl
rdkafka.ssl.ca.location      /tmp/ca.cert
rdkafka.ssl.certificate.location /tmp/cert.pem
rdkafka.ssl.key.location         /tmp/cert.key

[mplachter]

I will test again but when I did the same fluent-bit would start up and say something to the effect of these settings are ignored

[gabe-sorensen]

I'm using 1.0.4, but I was seeing this error when I was running:

[error] [out_kafka] cannot configure 'certificate.location' property

I forgot to add the .ssl before the property, which was my issue

[michaeltravisuk]

I'm still having trouble with this. Getting the following in my fluent-bit logs:

[2019/04/09 15:04:17] [error] [out_kafka] cannot configure 'security.protocol' property [2019/04/09 15:04:17] [error] [out_kafka] cannot configure 'ssl.ca.location' property [2019/04/09 15:04:17] [error] [out_kafka] cannot configure 'ssl.certificate.location' property [2019/04/09 15:04:17] [error] [out_kafka] cannot configure 'ssl.key.location' property

I've double checked the librdkafka parameters and made sure that all the referenced certificates/keys are valid and readable. This is on fluent-bit 1.0.6 (td-agent-bit).

[edsiper]

for anyone facing an issue please paste full configuration and fluent bit output

[michaeltravisuk]

fluent-bit (1.0.6) config is:

[SERVICE]
    Daemon       Off
    Parsers_File /etc/td-agent-bit/parsers.conf

[INPUT]
    Name        tcp
    Listen      0.0.0.0
    Port        5170
    Chunk_Size  32
    Buffer_Size 64

[FILTER]
    Name     parser
    Match    *
    Key_Name message
    Parser   logformat

[OUTPUT]
    Name Kafka
    Match *
    Topics {{topic-name}}
    Timestamp_Key time
    Brokers {{kafka-broker:9093}}
    rdkafka.request.required.acks 1
    rdkafka.log.connection.close false
    rdkafka.compression.codec snappy
    rdkafka.security.protocol    ssl
    rdkafka.ssl.ca.location /etc/td-agent-bit/kafka-ca.crt
    rdkafka.ssl.certificate.location /etc/td-agent-bit/kafka-client.crt
    rdkafka.ssl.key.location /etc/td-agent-bit/kafka-client.key

The output I get is:

Fluent Bit v1.0.6
Copyright (C) Treasure Data

[2019/04/10 11:08:42] [ info] [storage] initializing...
[2019/04/10 11:08:42] [ info] [storage] in-memory
[2019/04/10 11:08:42] [ info] [storage] normal synchronization mode, checksum disabled
[2019/04/10 11:08:42] [ info] [engine] started (pid=98108)
[2019/04/10 11:08:42] [error] Error binding socket
[2019/04/10 11:08:42] [ warn] Cannot listen on 0.0.0.0 port 5170
[2019/04/10 11:08:42] [error] [in_tcp] could not bind address 0.0.0.0:5170. Aborting
[2019/04/10 11:08:42] [error] Failed initialize input tcp.0
[2019/04/10 11:08:42] [error] [out_kafka] cannot configure 'security.protocol' property
[2019/04/10 11:08:42] [error] [out_kafka] cannot configure 'ssl.ca.location' property
[2019/04/10 11:08:42] [error] [out_kafka] cannot configure 'ssl.certificate.location' property
[2019/04/10 11:08:42] [error] [out_kafka] cannot configure 'ssl.key.location' property
[2019/04/10 11:08:42] [ info] [out_kafka] brokers='broker-name:9093' topics='topic.name'
[2019/04/10 11:08:43] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-broker:9093/bootstrap]: kafka-broker:9093/bootstrap: Receive failed: Connection reset by peer (after 3ms in state UP)
[2019/04/10 11:08:43] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-broker:9093/bootstrap]: kafka-broker:9093/bootstrap: Receive failed: Connection reset by peer (after 3ms in state UP)
[2019/04/10 11:08:43] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-broker:9093/bootstrap]: 1/1 brokers are down
[2019/04/10 11:08:44] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-broker:9093/bootstrap]: 1/1 brokers are down
[2019/04/10 11:08:45] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-broker:9093/bootstrap]: 1/1 brokers are down
[2019/04/10 11:08:46] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-broker:9093/bootstrap]: 1/1 brokers are down
[2019/04/10 11:08:47] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-broker:9093/bootstrap]: 1/1 brokers are down
[2019/04/10 11:08:48] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-broker:9093/bootstrap]: 1/1 brokers are down

[michaeltravisuk]

Done a bit more local testing, Got a basic setup capturing CPU stats and sending to a secured topic. The output config looks like this:

[OUTPUT]
  Name Kafka
  Match *
  Topics secure-logs
  Timestamp_Key time
  Brokers kafka-0:9093
  rdkafka.request.required.acks 1
  rdkafka.log.connection.close false
  rdkafka.compression.codec snappy
  rdkafka.security.protocol ssl
  rdkafka.ssl.ca.location /etc/fluent-bit/ca.crt
  rdkafka.ssl.certificate.location /etc/fluent-bit/client.crt
  rdkafka.ssl.key.location /etc/fluent-bit/client.key

On fluent-bit 0.13.7 this works correctly (no output is reported and the messages appear on the queue).

On fluent-bit 1.0.6 with the exact same config and file locations:

Fluent Bit v1.0.6
Copyright (C) Treasure Data

[2019/04/10 11:00:48] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-0:9093/bootstrap]: 1/1 brokers are down
[2019/04/10 11:00:49] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-0:9093/bootstrap]: 1/1 brokers are down
[2019/04/10 11:00:50] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-0:9093/bootstrap]: 1/1 brokers are down
[2019/04/10 11:00:51] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-0:9093/bootstrap]: 1/1 brokers are down

Update: Tried this against a real live Kafka cluster, can confirm that fluent-bit 0.13.7 works, but not fluent-bit-1.0.6

[michaeltravisuk]

Any update on this? We're still unable to use current versions of fluent-bit to output messages to our Kafka cluster. Any attempt to set librdkafka SSL configuration fails (other librdkafka config works:

Using fluent-bit 1.0.6 in debug mode:

[2019/05/20 10:10:55] [error] [out_kafka] cannot configure 'security.protocol' property
[2019/05/20 10:10:55] [error] [out_kafka] cannot configure 'ssl.ca.location' property

[michaeltravisuk]

I've managed to get it working, but I've had to compile the binary myself.
Even with the build option FLB_TLS set to On I had to also have the openssl-devel package installed (this is on CentOS - we run a combination of CentOS 6 and 7) for the resulting binary to actually allow me to send messages to an SSL fronted Kafka cluster.

Assuming that the td-agent-bit CentOS RPMs are built from:
https://github.com/fluent/fluent-bit-packaging/blob/master/distros/centos/7/Dockerfile.base

openssl-devel isn't included, and FLB_TLS isn't specified in:
https://github.com/fluent/fluent-bit-packaging/blob/master/distros/centos/7/Dockerfile#L12
It is is set to Off by default as I understand it.

[den-is]

so still no definitive fix to that issue?

Aug 03 15:22:46 ip-10-70-28-193.ec2.internal td-agent-bit[2849]: [2019/08/03 15:22:46] [error] [out_kafka] cannot configure 'security.protocol' property
...
[2019/08/03 19:04:52] [error] [out_kafka] fluent-bit#producer-1: [thrd:b-1.prod-logs.sqjyzw.c2.kafka.us-east-1.amazonaws.com:9094/]: 3/3 brokers are down
^C[engine] caught signal (SIGINT)
[2019/08/03 19:04:53] [error] [out_kafka] fluent-bit#producer-1: [thrd:b-1.prod-logs.sqjyzw.c2.kafka.us-east-1.amazonaws.com:9094/]: b-1.prod-logs.sqjyzw.c2.kafka.us-east-1.amazonaws.com:9094/bootstrap: Timed out 0 in-flight, 2 retry-queued, 0 out-queue, 0 partially-sent requests

i'm getting same'ysh issue while trying to use fluentbit with AWS MSK (managed kafka) which is TLS.
At the same time absolutely same config is working fine with fluentbit running in container in kubernetes cluster as daemon set.

Debug shows no additional informative messages.
No custom certificates or CAs. Standard public managed AWS service.
This is CentOS 7 and system has openssl-devel and ca-certificates installed.
Td-agent-bit installed via official RPM repo.
Fluent-bit td-agent-bit 1.2.2

[OUTPUT]
    Name             kafka
    Match            *
    Brokers          alteredworker1:9094,alteredworker2:9094,alteredworker39094
    Topics           logstash
    Timestamp_Key    @timestamp
    Timestamp_Format iso8601
    rdkafka.security.protocol          ssl
    rdkafka.log.connection.close       false
    rdkafka.request.required.acks      1
    rdkafka.queue.buffering.max.kbytes 10240

[edsiper]

thanks for the info.

Indeed the Dockerfiles build images used to package don't have openssl-devel (RPM) and libssl/libssl-dev on Debian based systems, so when librdkafka builds there is no SSL support.

Future release will come with proper dependency linked.

[lifeofguenter]

@edsiper is this still an issue? Seems like it is fixed?

[Hichem-BenJeddi]

I had the same issue with the old version. The latest version has fixed the issue now the SSL authentication with kafka is working correctly

[edsiper]

this is fixed, thanks for the follow up.

closing it.

[aderuwe]

This does not seem fixed in the packages for ubuntu in version 1.4.5 as I still get this error?

[deepankarmalhan]

Still running into this issue @edsiper for v1.5.0 as well. Running v1.3.2 in a container works (the docker image is built differently so that makes sense).

[paul-greidanus-hs]

I just submitted an issue against the packaging repo: https://github.com/fluent/fluent-bit-packaging/issues/13