Keda: milli values scale incorrectly

[agolomoodysaada]

I'm using the kafka ScaledObject. The HPA sees milli values incorrectly scaling them too high up.
In my case lagThreshold is set to "1". And the HPA sees the target as 1438m/1 (avg).
What's strange is that it's scaling up replicas to the max possible number based as though it's "1438/1" which is definitely incorrect!
Change the target from "1" to "1000" seems to fix the issue? This is definitely a bug though.

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: myapp
  labels:
    deploymentName: myapp # must be in the same namespace as the ScaledObject
spec:
  scaleTargetRef:
    deploymentName: myapp # must be in the same namespace as the ScaledObject
  pollingInterval: 10  # Optional. Default: 30 seconds
  cooldownPeriod:  10 # 60 # Optional. Default: 300 seconds
  minReplicaCount: 1   # Optional. Default: 0
  maxReplicaCount: 20 # Optional. Default: 100
  triggers:
  - type: kafka
    metadata:
      brokerList: kafka-kafka-headless.kafka:9092
      consumerGroup: myconsumergroup
      topic: mytopic
      lagThreshold: "1"

[yaron2]

Can you look at the logs of the keda controller and put the output here?
You should see messages like Group X has a lag of X for topic X and partition X

[jeffhollan]

Can you look at the logs of the keda controller and put the output here?
You should see messages like Group X has a lag of X for topic X and partition X

Note for this I think you'll need to modify the default KEDA config and set the log-level to debug as this doesn't show right now as info (though debatably it should). I'll also see if I can repro later today

[agolomoodysaada]

I'm using the helm chart but I don't see these logs. Here's my values file.

keda-edge:
  logLevel: debug
  glogLevel: 2

[jeli8-cor]

Hey guys, I think I can contribute more data to this issue. I'm facing the same issue with Kafka scaler right now. My numbers are a bit different, but the same problem is here.
Few points:

• kafka_scaler should print the message of "Group X has a lag of Y..." in info mode according to the code here but only when I changed the logs to debug I saw these prints.

• I have topic with 20 partitions but in the logs of keda-opertaor I can only see one print for partition 0:

{"level":"debug","ts":"2020-05-19T07:29:18.569Z","logger":"kafka_scaler","msg":"Group has a lag of 881 for topic gateway and partition 0\n"}

And that's the hpa metrics, running at the same time:

keda-hpa-worker Deployment/worker-logsparser 3481300m/7500 (avg)

I'm not sure why the metrics is with "m" or indicating millis...? But the lag for each partition is around that number (800 - 1000 msgs lag) and the number of min replicas I have is 10.
So, my guess was that the hpa formula gets the following:
currentReplicas = 10 , currentMetricValue = 881 , desiredMetricValue = 7500 ===>
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]
desiredReplicas = ceil[10 * (881/7500)] = 2 ==> meaning, we keep the min pods.
The desiredMetricValue is some workaround I did cause I couldn't figure out the calculation, but now it seems like a bug in the current metric value (by the Kafka scaler) cause it only sees 1 partition instead of 20....
Let me know if I can help with more or provide more details.

[jeli8-cor]

Another thing I noticed, not sure if it helps... The info message which should be print after sum up all partitions' lag is not displayed in keda-operator logs:

kafkaLog.V(1).Info(fmt.Sprintf("Kafka scaler: Providing metrics based on totalLag %v, partitions %v, threshold %v", totalLag, len(partitions), s.metadata.lagThreshold))

The log info can be found here

[jeli8-cor]

Hey guys,
Sorry for the multiple confusing messages but I think I finally got it - the hpa display the targetAverageValue which is the value it gets from Keda scaler (in Kafka case - total lag from all partitions) and divide it by the number of pods (not the num of partitions as I said before).
Bottom line, I tried to find a way to get num of pods from outside the scaler in order to calculate it with the result, before returning it, but couldn't find a way.
My idea is to return something like that:
newTotalLag = TotalLag + - (TotalLag modulu )
This will always result with a divisible by num of pods number and we won't see the milli in HPA anymore.
I'll be happy to make the change and open a PR, I just not sure where to get it from.
@zroubalik maybe you can assist with that?

[zroubalik]

@jeli8-cor pardon my ignorance. I can see where are you going with your latest comment. But what exactly is the problem you are trying to solve?

[zroubalik]

Another thing I noticed, not sure if it helps... The info message which should be print after sum up all partitions' lag is not displayed in keda-operator logs:

kafkaLog.V(1).Info(fmt.Sprintf("Kafka scaler: Providing metrics based on totalLag %v, partitions %v, threshold %v", totalLag, len(partitions), s.metadata.lagThreshold))

The log info can be found here

have you enabled debug logging on the Metrics server?

[jeli8-cor]

@zroubalik , I'm trying to solve the following issue where the HPA current metric (average) is appear sometimes with "m" at the end and multiplied by 1000
The following output is from running: kubectl get hpa -w
keda-hpa-enrich Deployment/enrich 17059500m/60k (avg), 7548500m/60k (avg) 1 5 2 4d23h

keda-hpa-enrich Deployment/enrich 28250/60k (avg), 28606/60k (avg) 1 5 2 4d23h
I enabled the debug logs, couldn't find this log message. I'll check again

[zroubalik]

@zroubalik , I'm trying to solve the following issue where the HPA current metric (average) is appear sometimes with "m" at the end and multiplied by 1000

And it is causing any problems? If so, what kind of problems?

I enabled the debug logs, couldn't find this log message. I'll check again

This log should apper in the Metrics Server log.

[zeroX-tj]

We are having the same issue with the "m" appended to the value, this makes our deployment scale up to the maximum number of pods

We didn't enable the logs yet but it seems to be a common issue

[zroubalik]

We are having the same issue with the "m" appended to the value, this makes our deployment scale up to the maximum number of pods

Thanks for the confirmation, wasn't sure if it is this problem.

@ppatierno wdyt?

[jeli8-cor]

@zroubalik ,that is what I tried to describe (poorly...). The issue with the "m" in HPA currentValue is a wide issue and not relevant to a specific scaler. I'm using the Kafka scaler so I thought about solving it there, but the issue is with all scalers, cause the value displayed in HPA (when you're running kubectl get hpa keda-hpa- -w) will be the value it got from the metrics server (in Kafka case - total lag for a given topic) divided by the num of current replicas (to display average value...). Thus, if your topic has a total lag of 100 message and you have 3 pods running, the value you'll see is: 33333m ---> which is 33.3 current lag per pod.
To solve that, I suggested - take the num of current replicas (if we have it somewhere) and add it to the current metric before sending it back.
Meaning, if (as in the example above)
total lag = 100, numOfPods = 3, =====> return totalLag = 102;

[Merlyn2020]

So the issue of the “m” is really a non-issue. The “m” is simply a way for Kubernetes HPA to show a qualitative custom metric value with an implicit decimal. 1.5 = 1500m that’s it.
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#appendix-quantities

Why did they choose this? Working with whole numbers is easier, you just divide by 1000. There may be some internationalization to it where some parts of the world still use a comma instead of a period as a decimal separator, probably less common now. They use a different approach by specifying numbers in a globally understood way, as just whole numbers with suffixes.

Also this “m” is specific to Kubernetes and has nothing to do with Keda or any custom metrics application.
So when debugging your scaling and you see an “m” in the target value make sure you divide it by 1000 so you use the same value that the HPA is using behind the scenes.

[brianpsr]

It still seems like there is a real issue here.

keda-hpa-rabbitmq-consumer Deployment/rabbitmq-consumer 9746600m/10k (avg) 1 30 5 2d

If you what you said true, then the real number the HPA is working with is 9746.6, and yet I have 5 pods. Now mind you 5
pods makes sense, because the real queuelength is around 49,000 right now, so the HPA is working properly. But I just
can't see how to correlate 9746600m to 49000. Divide it by 200 and we get close, but where is the logic in that?

I guess it is correct, but confusing to read. Number shown / 1000 / number of pods = avg number per pod heh.