Librdkafka: Segfault when Monitoring Interceptors are enabled

[agis]

Description

Note: I'm using confluent-kafka-go, but I've opened the issue here because I think this is related to librdkafka and not the Go driver. But I could be wrong, so let me know and I can transfer the issue there.

TL;DR When Monitoring Interceptors are enabled with librdkafka 1.1.x or 1.2.x (haven't tested 1.0.x), a segmentation fault occurs most of the time, when closing the producer.

We used the Monitoring Interceptor plugin from the confluent-librdkafka-plugins Debian package. Everything works fine with librdkafka 0.11.6. However upon upgrading to 1.1.0 and subsequently 1.2.0, we've experienced a segfault when we call producer.Close(). This happens with both confluent-librdkafka-plugins 0.11.0 and 0.11.1. As I said above, this happens most of the time, but not always.

I provide below a very simple sample code script below that reliably reproduces this.

librdkafka debug logs: https://gist.github.com/agis/b8e97eb0746c39f33e43acb1d7fd9407

How to reproduce

Install the Debian package (0.11.1 or 0.11.0):

$ sudo apt install confluent-librdkafka-plugins

Clone https://github.com/agis/confluent-interceptor-segfault and do:

$ go run main.go <broker> <topic>

It crashes on librdkafka 1.1.0, 1.2.0, 1.2.2-RC2 and 1.2.2:

$ go run main.go kafka.example.com:9092 test
16843007 1.1.0
signal: segmentation fault

$ LD_PRELOAD=/home/agis/dev/librdkafka/src/librdkafka.so.1 go run main.go kafka.example.com:9092 test
16909055 1.2.2-RC2
signal: segmentation fault

Checklist

• [x] librdkafka version (release number or git tag): 1.1.0, 1.2.0, 1.2.2-RC2, 1.2.2
• [x] Apache Kafka version: 1.1.1
• [x] librdkafka client configuration: plugin.library.paths=monitoring-interceptor
• [x] Operating system: Debian 10 (Buster) x86
• [x] Provide logs (with debug=.. as necessary) from librdkafka: https://gist.github.com/agis/b8e97eb0746c39f33e43acb1d7fd9407
• [ ] Provide broker log excerpts
• [ ] Critical issue

[agis]

@edenhill Hey, any updates on this one?

P.S. Asking just to decide if we should wait for the next release before upgrading, or just disable the interceptors for now.

[agis]

Confirmed that this is also an issue in the newly-released 1.2.2.

[edenhill]

I've now reproduced the issue, I believe it is caused by the interceptor inheriting the configuration from the intercepted client which has delivery report events enabled , while the interceptor client does not make use of delivery reports and has no callback set, which causes a null pointer dereference in its flush() call.

We'll get this fixed in the upcoming CP 5.4 release.

[agis]

We'll get this fixed in the upcoming CP 5.4 release.

@edenhill To clarify, does this mean a new confluent-librdkafka-plugins Debian package will be released in the Confluent repositories that fixes this? Currently it's at v0.11.1.

[edenhill]

@agis Yes, it will be v0.11.3