Keda: Initial readiness delay with metrics api and prometheus metrics

[recet]

The equivalent of the Kubernetes HPA --horizontal-pod-autoscaler-initial-readiness-delay flag that can be set by providing the parameter initialReadinessDelay in the ScaledObject spec YAML.

Managed kubernetes platforms (for example GKE) does not allow to modify --horizontal-pod-autoscaler-initial-readiness-delay flag. Unlike the HPA this does not need to be set as a flag for the kube-controller-manager on the master node, and can be deployment/statedulset specific.

A pod startup for slow applications (that need more resources at the start, for example CPU) should not create a cascaded scale up effect. We want to set/modify the initial delay before the metrics are considered for autoscaling for a given deployment/statefulset.

[silenceper]

Fundamentally, keda also creates an HPA object to control scaling(type: External).

I think that if the pod starts slowly, the indicator data may still maintain the previous state. In the worst case, it should only be a repeated setting of the replicas of workload (the replicas is still the same as the last time). //Maybe i'm wrong

[recet]

Not sure if I follow. To be precise, I was imagining a configuration at horizontalPodAutoscalerConfig.behavior.

For example this section from the doc:

advanced:
  horizontalPodAutoscalerConfig:                   # Optional. Section to specify HPA related options
    resourceMetrics:                               # Optional. If not set, KEDA won't scale based on resource utilization
    - name: cpu/memory                             # Name of the metric to scale on
      target:
        type: value/ utilization/ averagevalue
        value: 60                                  # Optional
        averageValue: 40                           # Optional
        averageUtilization: 50                     # Optional
    behavior:                                      # Optional. Use to modify HPA's scaling behavior
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
        - type: Percent
          value: 100
          periodSeconds: 15

Also contained initialReadinessDelay: 300 similar to stabilizationWindowSeconds. Does that makes sense?

[turbaszek]

@recet the initialReadinessDelay seems to be missing from what ScaledObjects support:
https://github.com/kedacore/keda/blob/f5489b8e229260817fdabcfb1b930bc38dba9337/config/crd/bases/keda.sh_scaledobjects.yaml#L70-L130

I think we may consider adding it? @zroubalik

[recet]

Thanks for the reply @turbaszek. Indeed the initialReadinessDelay is missing at the moment.

We have some applications that consume lots of resources on start-up (expected) and might need some extra time to startup, but we do not want it to scale up during this period. However, the default HPA behaviour scale these applications unnecessarily.

Adding this feature, if possible, would extend the default HPA behaviour and make this project even more attractive to us. That being said, I am not sure if this would be possible. Looking forward to a positive reply :smile:

[silenceper]

Maybe this parameter should be given to the hpa object setting in Kubernetes to be more appropriate.

like this https://github.com/kubernetes/kubernetes/pull/74525

[zroubalik]

I don't see any problem with adding this option.

@recet I was quickly looking for this property in HPA spec, but wasn't able to find it? Could you please link here some details on this property (link to the HPA spec field of this property).

And is this something, that you'd like to contribute? It should be just about adding this property to the created HPA resource, the same way it is done for the behavior field for example.

[recet]

Unfortunately, the --horizontal-pod-autoscaler-cpu-initialization-periodand the--horizontal-pod-autoscaler-initial-readiness-delay` flags are not configurable by the hpa spec today. ref

Not even sure if it is feasible then? Maybe this should be a request to kubernetes project directly?

[zroubalik]

Yeah, unfortunately that's not something that we can do from KEDA side, as we rely on HPA for the scaling part.

[recet]

thanks for the response @zroubalik. I guess we can close this issue in that case.

[recet]

This parameter should be given to the HPA object setting in Kubernetes as KEDA relies on HPA for the scaling part.