Fluent-bit: kafka is broken with new logging API

[edsiper]

On v1.4 Kafka plugin, new logging API is referring to an undefined context:

[engine] caught signal (SIGSEGV)
#0  0x55e06e0befe6      in  cb_kafka_msg() at plugins/out_kafka/kafka.c:38
#1  0x55e06e31c1a0      in  rd_kafka_poll_cb() at plugins/out_kafka/librdkafka-1.3.0/src/rdkafka.c:3459
#2  0x55e06e387b66      in  rd_kafka_op_handle() at plugins/out_kafka/librdkafka-1.3.0/src/rdkafka_op.c:681
#3  0x55e06e397482      in  rd_kafka_q_serve() at plugins/out_kafka/librdkafka-1.3.0/src/rdkafka_queue.c:485
#4  0x55e06e31c495      in  rd_kafka_poll() at plugins/out_kafka/librdkafka-1.3.0/src/rdkafka.c:3557
#5  0x55e06e0bfc5a      in  produce_message() at plugins/out_kafka/kafka.c:269
#6  0x55e06e0bfd78      in  cb_kafka_flush() at plugins/out_kafka/kafka.c:309
#7  0x55e06e04bf1c      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:371
#8  0x55e06e487e8a      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#9  0xffffffffffffffff  in  ???() at ???:0

[edsiper]

I did some troubleshooting and there are two possible causes: I am using the API in the wrong way or librdkafka is not passing the opaque data type properly, for the second case I've opened a ticket with librdkakfa folks:

https://github.com/edenhill/librdkafka/issues/2789

[edsiper]

For now, I am going to remove the need for the opaque data type.

[edsiper]

Fixed on f69246a3

[edsiper]

Fix will be part of v1.4.2 release

[AkshayDubey29]

@edsiper Thanks for the quick turn around.

[zebraincognito]

I have the same issue with fluent-bit v.1.4.4. Debug/trace are pretty useless so there is no chance to find out what happened from logs:

[2020/08/19 11:34:25] [debug] [output:kafka:kafka.0] enqueued message (83 bytes) for topic 'topicname'
[2020/08/19 11:34:25] [debug] [output:kafka:kafka.0] enqueued message (83 bytes) for topic 'topicname'
[2020/08/19 11:34:25] [debug] [output:kafka:kafka.0] enqueued message (83 bytes) for topic 'topicname'
[engine] caught signal (SIGSEGV)
#0  0x83f118            in  msgpack_pack_object() at lib/msgpack-3.2.0/src/objectc.c:95
#1  0x519703            in  produce_message() at plugins/out_kafka/kafka.c:154
#2  0x519edb            in  cb_kafka_flush() at plugins/out_kafka/kafka.c:318
#3  0x4be5b0            in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:371
#4  0x85ca25            in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#5  0xffffffffffffffff  in  ???() at ???:0

So it's unlikely was fixed or there are several reasons for that