Keda: Increased latency on EKS after Keda has been deployed

[idonca]

Hello,

We were facing some increase on the latency side because KEDA did calls to external.metrics.k8s.io. This is increasing response time on kubernetes api queries.
Do you know how we can "isolate" keda to watch and work only on some specific k8s namespaces?
Can we deactivate that API or can we limit the calls made to that endpoint? (external.metrics.k8s.io)

Or did you have this problem also?

Attached are some responses from k8s APIs:

api-response.txt
pods-verbose.txt

Thank you,
-Ionut

[idonca]

Hello,

Any help with this problem?

Thanks

[tomkerkhove]

I haven't seen any reports on this but maybe @zroubalik can help in terms of the API server

[zroubalik]

Hi, interesting, I have never seen this. You can limit namespaces watched by KEDA by setting env WATCH_NAMESPACE variable on both Operator and Metrics Server.
Is the scaling working correctly for you? Have you seen another errors somewhere around the cluster?

[idonca]

Hello,

Thanks for the reply.
That is a cool feature to only tell KEDA to look on only one namespace.

Yeah, the scaling is working correctly, the problem is this: in that eks cluster we run some automation tests that are triggering a lot of pods and new namespaces each day. Maybe that was the problem.

I am still looking and investigating, I uninstalled the keda helm chart and installed again and now the latency on the api is looking good.

Maybe the latency is increasing after a period of time.

Can you please leave this issue open till next week when I will come back with more details?

Thank you very much,
-Ionut

[tomkerkhove]

We can definately do that! If you have suggestions to improve our docs on this as well, we are happy to welcome PRs on https://github.com/kedacore/keda-docs

[zroubalik]

Absolutely, happy to hear your findings.