Librdkafka: Excessive TCP connections by simple consumer

Created on 15 Apr 2020 · 6Comments · Source: edenhill/librdkafka

Description

Clarification: simple consumer is the consumer that consumes from a fixed set of partitions through the assign API (vs. the consumer that consumes messages as a member of a Kafka managed consumer group).

The simple consumer is making a connection to all partition leaders, even those that are not in its assigned partitions. This is different from how Java simple consumer behaves.

How to reproduce

When I run the Java simple consumer (e.g. bin/kafka-console-consumer.sh --bootstrap-server <kafkabroker>:9092 --topic test --partition 0), I see that a single connection is made from the consumer node to the bootstrap-server (<kafkabroker>) and one more connection to the leader of partition test(0) (2 connections in total). I verify number of connections by running netstat -a | grep <kafkabroker_ip> | wc -l from the consumer node.

However, unless I'm missing something, for librdkafka things seem to be quite different. What I did was I simply modified the file examples/rdkafka_complex_consumer_example.cpp (release 1.4.0) and replaced the consumer subscription call to a simple consumer assignment call:

RdKafka::ErrorCode err = consumer->subscribe(topics);

"replaced by"

std::vector<RdKafka::TopicPartition*> partitions;
partitions.push_back(RdKafka::TopicPartition::create(topics[0], 0));
RdKafka::ErrorCode err = consumer->assign(partitions);

After running this modified consumer (e.g. examples/rdkafka_complex_consumer_example_cpp -g test -b <kafkabroker>:9092 test), I see that only messages from partition 0 are being consumed. However, I see the consumer node is making the following steady connections:

1 connection to bootstrap server
1 connection to the group coordinator (even though the group remains empty during the consumption)
1 connection to every broker that is a leader of test partition (if topic test has 60 partitions uniformly spread across 10 brokers, this adds up to 10 connections)

We have seen this issue in one of our production clusters that hosts thousands of partitions and consumed by librdkafka client. We are observing the number of TCP connections to the cluster far above that of clusters that are used by Java clients only.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

[x] librdkafka version (release number or git tag): 1.4.0
[x] Apache Kafka version: 2.3.1
[x] librdkafka client configuration: default configs
[x] Operating system: Consumer running on Mac or Ubunto 14, Cluster running on Ubuntu 14
[ ] Provide logs (with debug=.. as necessary) from librdkafka
[ ] Provide broker log excerpts
[ ] Critical issue

GREAT REPORT bug consumer

Source

vahidhashemian

Most helpful comment

The bootstrap and group coordinator connections are expected (the group coord connection is needed for offset commits and is considered a persistent connection if group.id is configured).
As for connection to leaders of partitions that are not being assigned, that does sound odd indeed and that should not happen.

I'll try to reproduce this.