Keda: Kafka scaler for multiple topics

[alexakr]

I would like to be able to defined Kafka trigger for multiple topics.

Use-Case

Kafka consumer application can subscribe to multiple topics and process data from them.
Kafka trigger will process offset lags of a list of topics and lagThreshold will be based on an aggregation of all the lags.

Specification

triggers:
metadata:
topics:
- topic1
- topic2
...

[bharathguvvala]

We, internally within our company, are using keda for various scalers it provides with kafka scaler being one of them. This is a perfectly valid use case where a workload can consume events from multiple topics (eg: events sequestered based on priorities p0, p1, p2 etc). The current kafka scaler spec is modeled to work with only 1 topic and adding multiple kafka scaler triggers to scaled object wouldn't work because of two reasons:

1. The triggers are evaluated independently, so providing separate triggers when the workload is processing events from all of them together is unintuitive. Further the user can't be expected to provide a cumulative lag_threshold for all the topics across many kafka triggers is also not reasonable.
2. The replica count for each trigger is capped at the number of partitions and the final target replicas will be max(trigger1, trigger2 ...) and not an aggregate of all of the resulting replica counts per trigger which is what it should be.

So the scaler spec should look like this :

- type: kafka
    metadata:
      bootstrapServers: localhost:9092
      consumerGroup: my-group
      eventSources:
      - topic: test-topic-1
        lagThreshold: "50"
     -  topic: test-topic-2       
        lagThreshold: "50"

I am willing to contribute to this feature once we discuss the validity of this feature and the solution.

NOTE: The scope of this proposal is to only scale out the pods only and any reassignment of consumers across the final set of pods is outside the purview of the scaler and totally upto the application.

[zroubalik]

I can see the need for multiple kafka topics per ScaledObject. I am more inclined to implement this be specifing multiple scalers/triggers per ScaledObject rather than modifing existing Kafka scaler. We would like to support this for upcoming KEDA v2.
Multiple scalers support: https://github.com/kedacore/keda/issues/733
For the replica count, you can see this proposal: https://github.com/kedacore/keda/issues/373

[alexakr]

I agree with @bharathguvvala spec proposal. This will eliminate configuration redundancy and hence reduce configuration mistakes. So the full spec including the authentication will look like this:

- type: kafka
    metadata:
      bootstrapServers: localhost:9092
      consumerGroup: my-group
      eventSources:
      - topic: test-topic-1
        lagThreshold: "50"
      - topic: test-topic-2       
        lagThreshold: "50"
    authenticationRef:
      name: keda-trigger-auth-kafka-credential

[alexandery]

I agree with the need for this. In my case I may need up to 20-30 topics (maybe even more) to monitor and scale deployments. As long as I don't need to create 20-30 scaled objects and can create one - that would be very helpful.