Keda: Cannot scale deployment with Prometheus and CPU scalers at the same moment

[leonfibal]

Expected Behavior

A deployment is scaled according to both scalers

Actual Behavior

We have configuration as below:

    - metadata:
        metricName: http_requests_total
        query: 'sum(rate(http_requests_total{job="MyJob"}[2m]))'
        serverAddress: 'http://prometheus'
        threshold: '3'
      type: prometheus
    - metadata:
        type: Utilization
        value: '5'
      type: cpu

Keda-operator-metrics-apiserver reports following error:

E1028 08:16:10.241821       1 wrap.go:39] apiserver panic'd on GET /apis/external.metrics.k8s.io/v1beta1/namespaces/namespace123/prometheus-http---prometheus-http_requests_total?labelSelector=scaledObjectName%3Ddeployment-api
2020-10-28 08:16:10.241892 I | http2: panic serving 10.221.5.102:36258: runtime error: invalid memory address or nil pointer dereference
goroutine 24730 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc000c84840)
        /go/pkg/mod/k8s.io/[email protected]/pkg/server/filters/timeout.go:108 +0x113
panic(0x22111e0, 0x3966910)
        /usr/local/go/src/runtime/panic.go:969 +0x175
github.com/kedacore/keda/pkg/provider.(*KedaProvider).GetExternalMetric(0xc000848050, 0xc000d421b1, 0xd, 0x29d7720, 0xc0009524c0, 0xc000d421bf, 0x45, 0x59041c0981a, 0x3994d20, 0x2595ffe)
        /workspace/pkg/provider/provider.go:92 +0x523
github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/external_metrics.(*REST).List(0xc000130d20, 0x29be060, 0xc0009843c0, 0xc000993f80, 0x0, 0x0, 0x297e260, 0xc000993f80)
        /go/pkg/mod/github.com/kubernetes-incubator/[email protected]/pkg/registry/external_metrics/reststorage.go:80 +0x142
k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1(0x29b7120, 0xc000972398, 0xc000570a00)
        /go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/handlers/get.go:269 +0xcd2
github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResource.func1(0xc000984300, 0xc0004670a0)
        /go/pkg/mod/github.com/kubernetes-incubator/[email protected]/pkg/apiserver/installer/installer.go:288 +0x91
k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc000984300, 0xc0004670a0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/metrics/metrics.go:323 +0x255
github.com/emicklei/go-restful.(*Container).dispatch(0xc0008cdc20, 0x7fbd866ed5d8, 0xc000c86900, 0xc000570a00)
        /go/pkg/mod/github.com/emicklei/[email protected]+incompatible/container.go:288 +0xa84
github.com/emicklei/go-restful.(*Container).Dispatch(...)
        /go/pkg/mod/github.com/emicklei/[email protected]+incompatible/container.go:199
k8s.io/apiserver/pkg/server.director.ServeHTTP(0x258e03c, 0x16, 0xc0008cdc20, 0xc0004c45b0, 0x7fbd866ed5d8, 0xc000c86900, 0xc000570a00)
        /go/pkg/mod/k8s.io/[email protected]/pkg/server/handler.go:146 +0x4bc
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fbd866ed5d8, 0xc000c86900, 0xc000570a00)
        /go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filters/authorization.go:64 +0x523
net/http.HandlerFunc.ServeHTTP(0xc000790f00, 0x7fbd866ed5d8, 0xc000c86900, 0xc000570a00)
        /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fbd866ed5d8, 0xc000c86900, 0xc000570a00)
        /go/pkg/mod/k8s.io/[email protected]/pkg/server/filters/maxinflight.go:160 +0x4d4
net/http.HandlerFunc.ServeHTTP(0xc000331f50, 0x7fbd866ed5d8, 0xc000c86900, 0xc000570a00)
        /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fbd866ed5d8, 0xc000c86900, 0xc000570a00)
        /go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filters/impersonation.go:50 +0x1fd2
net/http.HandlerFunc.ServeHTTP(0xc000790f40, 0x7fbd866ed5d8, 0xc000c86900, 0xc000570a00)
        /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fbd866ed5d8, 0xc000c86900, 0xc000570000)
        /go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filters/authentication.go:70 +0x652
net/http.HandlerFunc.ServeHTTP(0xc000849900, 0x7fbd866ed5d8, 0xc000c86900, 0xc000570000)
        /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc000c84840, 0xc00048a460, 0x29c01e0, 0xc000c86900, 0xc000570000)
        /go/pkg/mod/k8s.io/[email protected]/pkg/server/filters/timeout.go:113 +0xb8
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
        /go/pkg/mod/k8s.io/[email protected]/pkg/server/filters/timeout.go:99 +0x1cc

When we enable only one scaler (no matter prometheus or cpu) everything is ok.

Specifications

• KEDA Version: 2.0.0-rc2
• Platform & Version: Ubuntu 16.04.6 LTS
• Kubernetes Version: 1.16.9
• Scaler(s): Prometheus + CPU

[bwkerbow]

It sounds like this is being worked on, but want to chime in to say I am experiencing same issue. Thanks for PR, silenceper.