Eventing: Kafka Channel Throughput Constrained to 5 msg per second

[ntzlqx]

Describe the bug
We are using a default broker backed by a Kafka channel. We have 20+ subscribers to the channel and I am load testing only two of them.
The scenario is simple - I am generating thousands of events targeting subscriber A. Subscriber A does some work (saves the event to the DB) and turns around and sends an event to subscriber B.
Subscribers A and B are ksvscs with respective triggers.
What we are observing is that despite we are able to enqueue thousands or requests through the broker, we receive back a throughput of about 5 requests per second.
I do not see more than 5 individual instances activated for subscriber A and less than that for subscriber B.

Expected behavior
Increased throughput up to a known resource constraint.

1. Identify the resource constraint - in our case this seems to be the channel dispatcher but this an educated guess
2. Ability to scale the constrained resource to a known degree
3. Observe the Kafka topic depth of unprocessed messages

To Reproduce

1. Configure two ksvc with respective triggers
2. Configure containerConcurrency: 1 and autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
3. Add some sleep (processing time) 400ms in each ksvcs
4. Send a cloud event after processing from subscriber A to subscriber B
5. Generate 50K messages targeting subscriber A

Knative release version
0.18

Additional context
Kafka cluster configuration - 3 partitions - default
Single default broker backed by a Kafka channel

[lionelvillard]

@ntzlqx about 5 rps sounds right. By default the Kafka dispatcher waits for the outgoing request to finish before processing the next message (strict ordering). Since you have 3 partitions and a processing time of 400ms, you should get 7.5 rps.

An easy fix is to increase the number of partitions.

What kind of guarantees are you expecting from Kafka, in terms of message ordering, duplicated messages, and dropped events?

[ntzlqx]

for this scenario I am looking for best effort - ordering is not important - but I have event types that I need ordering - is the best way to achieve this to define additional brokers with respective Kafka channels (topics) and change the QoS per broker-channel-topic-combo?

[ntzlqx]

also I am seeing that different type of events are triggering despite there is a large queue of the load tested event type...and I have a single topic - how is this achieved - the other events somehow get on different partition? how do you control the partition distribution of event types - topics feels like the right answer but what is the right abstraction to achieve this - topic per broker? What's puzzling is that I have a single broker-channel-topic

[ntzlqx]

Happy to report that scaling the partitions to 1000 across 10 kafka brokers and 10 zookeepers with 10 channel dispatchers did the trick. Now the database is maxed out. @lionelvillard thank you very much.
Question:

1. Are there recommendations about the number of channel dispatchers, kafka brokers and zookeepers?
2. Number of topics/channels
   Thank you very much for all your help!

[lionelvillard]

@ntzlqx I opened an issue there since it is Kafka-specific.