Keda: azure-servicebus scaler erroring with blank error

Created on 11 Jan 2021 · 8Comments · Source: kedacore/keda

I am using the latest KEDA v2 release and after setting up my scaled function and scaled object it outputs every 30 seconds a very non-descriptive error into the keda-operator container logs. My configuration as far as I'm aware is correct and even switching KEDA to use debug logging it still gives no useful information to help me solve this issue.

Expected Behavior

The scaler should work and not throw errors every 30 seconds, or at least output a meaningful error if I have broken the configuration somewhere

Actual Behavior

The scaler is erroring with a blank error message

Steps to Reproduce the Problem

Add an Azure Function that uses a service bus subscription
Add a scaled object with the correct parameters
It fails to scale correctly

Logs from KEDA operator

2021-01-11T21:56:46.388585819Z  2021-01-11T21:56:46.388Z    ERROR   azure_servicebus_scaler error   {"error": ""}
2021-01-11T21:56:46.388640720Z  github.com/go-logr/zapr.(*zapLogger).Error
2021-01-11T21:56:46.388652220Z      /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
2021-01-11T21:56:46.388660020Z  github.com/kedacore/keda/pkg/scalers.(*azureServiceBusScaler).IsActive
2021-01-11T21:56:46.388668920Z      /workspace/pkg/scalers/azure_servicebus_scaler.go:134
2021-01-11T21:56:46.388676420Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScaledObjectScalers
2021-01-11T21:56:46.388683720Z      /workspace/pkg/scaling/scale_handler.go:203
2021-01-11T21:56:46.388688520Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScalers
2021-01-11T21:56:46.388694320Z      /workspace/pkg/scaling/scale_handler.go:192
2021-01-11T21:56:46.388699120Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).startScaleLoop
2021-01-11T21:56:46.388703620Z      /workspace/pkg/scaling/scale_handler.go:135
2021-01-11T21:56:46.388708320Z  2021-01-11T21:56:46.388Z    DEBUG   scalehandler    Error getting scale decision    {"Error": ""}
2021-01-11T21:56:47.341918395Z  2021-01-11T21:56:47.341Z    ERROR   azure_servicebus_scaler error   {"error": ""}
2021-01-11T21:56:47.341970495Z  github.com/go-logr/zapr.(*zapLogger).Error
2021-01-11T21:56:47.341981895Z      /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
2021-01-11T21:56:47.341991695Z  github.com/kedacore/keda/pkg/scalers.(*azureServiceBusScaler).IsActive
2021-01-11T21:56:47.341999495Z      /workspace/pkg/scalers/azure_servicebus_scaler.go:134
2021-01-11T21:56:47.342007396Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScaledObjectScalers
2021-01-11T21:56:47.342013196Z      /workspace/pkg/scaling/scale_handler.go:203
2021-01-11T21:56:47.342020596Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScalers
2021-01-11T21:56:47.342026896Z      /workspace/pkg/scaling/scale_handler.go:192
2021-01-11T21:56:47.342033296Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).startScaleLoop
2021-01-11T21:56:47.342038996Z      /workspace/pkg/scaling/scale_handler.go:135
2021-01-11T21:56:47.342044596Z  2021-01-11T21:56:47.341Z    DEBUG   scalehandler    Error getting scale decision    {"Error": ""}
2021-01-11T21:56:47.346960941Z  2021-01-11T21:56:47.346Z    DEBUG   scaleexecutor   ScaleTarget no change   {"scaledobject.Name": "attachment-thmbgn-funcs-scaledobject", "scaledObject.Namespace": "attachment", "scaleTarget.Name": "attachment-thmbgn-funcs-scaled"}

Specifications

KEDA Version: docker.io/kedacore/keda:2.0.0
Platform & Version: Linux, Azure Functions v3
Kubernetes Version: v.19.3
Scaler(s): azure-servicebus

Definitions

Scaled Object:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: attachment-thmbgn-funcs-scaledobject
spec:
  scaleTargetRef:
    name: attachment-thmbgn-funcs-scaled
  triggers:
  - metadata:
      connectionFromEnv: AzureServiceBusPrimary
      subscriptionName: images-function
      topicName: attachments
    type: azure-servicebus
  - metadata:
      connectionFromEnv: AzureServiceBusPrimary
      subscriptionName: pdf-function
      topicName: attachments
    type: azure-servicebus

Deployment:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: attachment-thmbgn-funcs-scaled
  labels:
    env: pr-7095
spec:
  replicas: 0
  selector:
    matchLabels:
      app: attachment-thmbgn-funcs-scaled
  template:
    metadata:
      labels:
        app: attachment-thmbgn-funcs-scaled
        env: pr-7095
    spec:
      containers:
        - name: attachment-thmbgn-funcs-scaled
          image: <censored>
          ports:
            - containerPort: 80
              protocol: TCP
          env:
            - name: APPINSIGHTS_INSTRUMENTATIONKEY
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: APPINSIGHTS_INSTRUMENTATIONKEY
            - name: AttachmentsConnectionString
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: AttachmentsConnectionString
            - name: AzureServiceBusPrimary
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: AzureServiceBusPrimary
            - name: AzureStorageBlob
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: AzureStorageBlob
            - name: AzureWebJobsStorage
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: AzureWebJobsStorage
            - name: ASPNETCORE_ENVIRONMENT
              value: pr-7095
            - name: ASPNETCORE_FORWARDEDHEADERS_ENABLED
              value: 'true'
            - name: AzureFunctionsJobHost__functions__0
              value: ImageThumbnailGenerator
            - name: AzureFunctionsJobHost__functions__1
              value: PdfThumbnailGenerator
            - name: AttachmentImageHeight
              value: '250'
            - name: AttachmentImageWidth
              value: '250'
            - name: BlobContainer
              value: attachments
            - name: FUNCTIONS_EXTENSION_VERSION
              value: ~3
            - name: FUNCTIONS_WORKER_RUNTIME
              value: dotnet
            - name: ImageHeight
              value: '250'
            - name: ImageSubscription
              value: images-function
            - name: ImageWidth
              value: '250'
            - name: ImagesContainer
              value: images
            - name: PdfSubscription
              value: pdf-function
            - name: ThumbnailBlobContainer
              value: thumbnails
            - name: TopicName
              value: attachments
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 25m
              memory: 200Mi
          livenessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            initialDelaySeconds: 60
            timeoutSeconds: 5
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            timeoutSeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
  strategy:
    type: Recreate

Secret (censored of course):

kind: Secret
apiVersion: v1
metadata:
  name: attachment-thmbgn-funcs-secret
data:
  APPINSIGHTS_INSTRUMENTATIONKEY: <censored-base64-value>
  AttachmentsConnectionString: >-
    <censored-base64-value>
  AzureServiceBusPrimary: >-
    <censored-base64-value>
  AzureStorageBlob: >-
    <censored-base64-value>
  AzureWebJobsStorage: >-
    <censored-base64-value>
type: Opaque

If you need any more information from me please do let me know, but I hope this is everything you need. Please note I have modified the definitions/logs very slightly just to remove sensitive company information but it all matches up.

azure bug scaler-azure-service-bus

Source

MattJeanes

Most helpful comment

I'm afraid I'll need @ahmelsayed for this

tomkerkhove on 19 Feb 2021

👍2

All 8 comments

Thanks for reporting, we'll get to the bottom of this.

@ahmelsayed can you pick this up?

tomkerkhove on 12 Jan 2021

❤1

Hi @tomkerkhove / @ahmelsayed, I'm still getting this issue and it's blocking us using KEDA for "serverless" functions on AKS, is there any information I can provide to help out?

MattJeanes on 19 Feb 2021

I'm afraid I'll need @ahmelsayed for this

tomkerkhove on 19 Feb 2021

👍2

sorry for missing this. taking a look now

ahmelsayed on 19 Feb 2021

@MattJeanes can you try updating the --zap-log-level=info arg on KEDA deployment to --zap-log-level=debug and share the logs?

ahmelsayed on 19 Feb 2021

@ahmelsayed here is some output logs, note there are a bunch of scalers - one of them at least is working fine which is the nginx scaler, but the others under "test-app" are not. I believe the 'no blobContainerName given' error is likely on me as well, it's primarily the Error getting scale decision on the azure-servicebus triggers which I am trying to investigate here:

2021-02-19T20:21:24.058Z    DEBUG   scalehandler    Error getting scale decision    {"Error": ""}
2021-02-19T20:21:24.058Z    DEBUG   scaleexecutor   ScaleTarget no change   {"scaledobject.Name": "test-app-crewappops-funcs-scaler", "scaledObject.Namespace": "test-app", "scaleTarget.Name": "test-app-crewappops-funcs-scaled"}
2021-02-19T20:21:24.484Z    INFO    controllers.ScaledObject    Reconciling ScaledObject    {"ScaledObject.Namespace": "ingress-nginx", "ScaledObject.Name": "ingress-nginx-controller"}
2021-02-19T20:21:24.484Z    DEBUG   controllers.ScaledObject    Parsed Group, Version, Kind, Resource   {"ScaledObject.Namespace": "ingress-nginx", "ScaledObject.Name": "ingress-nginx-controller", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
2021-02-19T20:21:24.484Z    DEBUG   scalehandler    cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped    {"type": "ScaledObject", "namespace": "ingress-nginx", "name": "ingress-nginx-controller", "env-var-name": "POD_NAME"}
2021-02-19T20:21:24.484Z    DEBUG   scalehandler    cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped    {"type": "ScaledObject", "namespace": "ingress-nginx", "name": "ingress-nginx-controller", "env-var-name": "POD_NAMESPACE"}
2021-02-19T20:21:24.485Z    DEBUG   controllers.ScaledObject    All metric names are unique in ScaledObject {"ScaledObject.Namespace": "ingress-nginx", "ScaledObject.Name": "ingress-nginx-controller", "value": "ingress-nginx-controller"}
2021-02-19T20:21:24.485Z    DEBUG   scalehandler    cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped    {"type": "ScaledObject", "namespace": "ingress-nginx", "name": "ingress-nginx-controller", "env-var-name": "POD_NAME"}
2021-02-19T20:21:24.485Z    DEBUG   scalehandler    cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped    {"type": "ScaledObject", "namespace": "ingress-nginx", "name": "ingress-nginx-controller", "env-var-name": "POD_NAMESPACE"}
2021-02-19T20:21:24.498Z    DEBUG   controllers.ScaledObject    ScaledObject is defined correctly and is ready for scaling  {"ScaledObject.Namespace": "ingress-nginx", "ScaledObject.Name": "ingress-nginx-controller"}
2021-02-19T20:21:24.510Z    DEBUG   controller  Successfully Reconciled {"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "name": "ingress-nginx-controller", "namespace": "ingress-nginx"}
2021-02-19T20:21:24.672Z    ERROR   azure_servicebus_scaler error   {"error": ""}
github.com/go-logr/zapr.(*zapLogger).Error
    /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/kedacore/keda/v2/pkg/scalers.(*azureServiceBusScaler).IsActive
    /workspace/pkg/scalers/azure_servicebus_scaler.go:137
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScaledObjectScalers
    /workspace/pkg/scaling/scale_handler.go:206
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
    /workspace/pkg/scaling/scale_handler.go:195
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
    /workspace/pkg/scaling/scale_handler.go:138
2021-02-19T20:21:24.673Z    DEBUG   scalehandler    Error getting scale decision    {"Error": ""}
2021-02-19T20:21:24.673Z    DEBUG   scaleexecutor   ScaleTarget no change   {"scaledobject.Name": "test-app-adsbtracking-funcs-scaler", "scaledObject.Namespace": "test-app", "scaleTarget.Name": "test-app-adsbtracking-funcs-scaled"}
2021-02-19T20:21:24.694Z    DEBUG   scaleexecutor   ScaleTarget no change   {"scaledobject.Name": "test-app-flightstatus-funcs-scaler", "scaledObject.Namespace": "test-app", "scaleTarget.Name": "test-app-flightstatus-funcs-scaled"}
2021-02-19T20:21:24.888Z    DEBUG   scaleexecutor   ScaleTarget no change   {"scaledobject.Name": "test-app-atlas-funcs-scaler", "scaledObject.Namespace": "test-app", "scaleTarget.Name": "test-app-atlas-funcs-scaled"}
2021-02-19T20:21:25.372Z    DEBUG   scalehandler    cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped    {"type": "ScaledObject", "namespace": "ingress-nginx", "name": "ingress-nginx-controller", "env-var-name": "POD_NAME"}
2021-02-19T20:21:25.372Z    DEBUG   scalehandler    cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped    {"type": "ScaledObject", "namespace": "ingress-nginx", "name": "ingress-nginx-controller", "env-var-name": "POD_NAMESPACE"}
2021-02-19T20:21:25.407Z    DEBUG   scaleexecutor   ScaleTarget no change   {"scaledobject.Name": "test-app-crewapp-funcs-scaler", "scaledObject.Namespace": "test-app", "scaleTarget.Name": "test-app-crewapp-funcs-scaled"}
2021-02-19T20:21:25.414Z    DEBUG   scaleexecutor   ScaleTarget no change   {"scaledobject.Name": "test-app-ops-funcs-scaler", "scaledObject.Namespace": "test-app", "scaleTarget.Name": "test-app-ops-funcs-scaled"}
2021-02-19T20:21:25.488Z    DEBUG   scalehandler    Scaler for scaledObject is active   {"Metrics Name": "prometheus-http---prometheus-kube-prometheus-prometheus-monitoring-svc-9090-promhttp_metric_handler_requests_total"}
2021-02-19T20:21:42.603Z    INFO    controllers.ScaledObject    Reconciling ScaledObject    {"ScaledObject.Namespace": "test-app", "ScaledObject.Name": "test-app-fuelpriceupload-funcs-scaler"}
2021-02-19T20:21:42.603Z    DEBUG   controllers.ScaledObject    Parsed Group, Version, Kind, Resource   {"ScaledObject.Namespace": "test-app", "ScaledObject.Name": "test-app-fuelpriceupload-funcs-scaler", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
2021-02-19T20:21:42.604Z    ERROR   controllers.ScaledObject    Unable to fetch scalers in metric name uniqueness check {"ScaledObject.Namespace": "test-app", "ScaledObject.Name": "test-app-fuelpriceupload-funcs-scaler", "error": "error getting scaler for trigger #0: error parsing azure blob metadata: no blobContainerName given"}
github.com/go-logr/zapr.(*zapLogger).Error
    /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).validateMetricNameUniqueness
    /workspace/controllers/scaledobject_controller.go:246
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/scaledobject_controller.go:193
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/scaledobject_controller.go:157
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
2021-02-19T20:21:42.604Z    ERROR   controllers.ScaledObject    Error checking metric name uniqueness   {"ScaledObject.Namespace": "test-app", "ScaledObject.Name": "test-app-fuelpriceupload-funcs-scaler", "error": "error getting scaler for trigger #0: error parsing azure blob metadata: no blobContainerName given"}
github.com/go-logr/zapr.(*zapLogger).Error
    /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/kedacore/keda/v2/controllers.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/scaledobject_controller.go:160
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
2021-02-19T20:21:42.617Z    ERROR   controller  Reconciler error    {"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "name": "test-app-fuelpriceupload-funcs-scaler", "namespace": "test-app", "error": "error getting scaler for trigger #0: error parsing azure blob metadata: no blobContainerName given"}
github.com/go-logr/zapr.(*zapLogger).Error
    /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:246
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90