Keda: Unable to get Azure Service Bus subscription ActiveMessageCount

[aminebizid]

My deployment don't scale because no metrics from Azure Service Bus

k -n hpa get hpa
NAME                REFERENCE             TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-consumer   Deployment/consumer   <unknown>/80%   1         10        1          3m

No errors on keda

k -n keda logs -f keda-keda-edge-584dd6ccc6-km8j6
time="2019-05-06T20:24:00Z" level=info msg="Deleted HPA with namespace hpa and name keda-hpa-consumer"
time="2019-05-06T20:24:13Z" level=info msg="Watching ScaledObject: hpa/azure-servicebus-queue-scaledobject"
time="2019-05-06T20:24:13Z" level=info msg="Created HPA with namespace hpa and name keda-hpa-consumer"

My bus

• topicName: testk8s
• subscriptionName: testk8ssub
• I created a policy called demorule under the topic with Manage rights

My secret

export SERVICEBUS_CONNECTION_STRING="Endpoint=sb://xxx.servicebus.windows.net/;SharedAccessKeyName=demorule;SharedAccessKey=xxx;EntityPath=testk8s"

k -n hpa create secret generic servicebuskey --from-literal=sb-connection-string=$SERVICEBUS_CONNECTION_STRING

My consumer deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: consumer
spec:
  selector:
    matchLabels:
      app: consumer
  template:
    metadata:
      labels:
        app: consumer
    spec:
      containers:
      - name: consumer
        image: aminrbizid/servicebus-topic:1.0.0
        args: ["testk8s", "testk8ssub"]
        env:
          - name: SERVICEBUS_CONNECTION_STRING
            valueFrom:
              secretKeyRef:
                name: servicebuskey
                key: sb-connection-string

My ScaledObject

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: azure-servicebus-queue-scaledobject
  namespace: hpa
spec:
  scaleTargetRef:
    deploymentName: consumer
  pollingInterval: 5
  minReplicaCount: 1   # Optional. Default: 0
  maxReplicaCount: 10 # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      # queueName: functions-sbqueue
      # or
      topicName: testk8s
      subscriptionName: testk8ssub
      # Required
      connection: SERVICEBUS_CONNECTION_STRING
      # Optional
      queueLength: "30" # default 5

Could you please give me the az command you use to get the connection string.
The connection string I use works with my consumer.

Could you please also add the error in logs if keda is unable to get the ActiveMessageCount.

[lee0c]

~I believe you want to set the connection value in your ScaledObject to the secret key, not the name of the container's environment variable.~ This is incorrect, failure to connect here is due to a bug & not an incorrect connection value.

Additionally, ScaledObjects need a label that sets the deploymentName:

metadata:
  name: azure-servicebus-queue-scaledobject
  label:
    deploymentName: consumer

[aminebizid]

@lee0c I did the changes as you mentionned
I tried

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: azure-servicebus-queue-scaledobject
  namespace: hpa
  label:
    deploymentName: consumer
spec:
  scaleTargetRef:
    deploymentName: consumer
  pollingInterval: 5
  minReplicaCount: 1   # Optional. Default: 0
  maxReplicaCount: 10 # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      # queueName: functions-sbqueue
      # or
      topicName: testk8s
      subscriptionName: testk8ssub
      # Required
      connection: servicebuskey
      # Optional
      queueLength: "30" # default 5

then

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: azure-servicebus-queue-scaledobject
  namespace: hpa
  label:
    deploymentName: consumer
spec:
  scaleTargetRef:
    deploymentName: consumer
  pollingInterval: 5
  minReplicaCount: 1   # Optional. Default: 0
  maxReplicaCount: 10 # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      # queueName: functions-sbqueue
      # or
      topicName: testk8s
      subscriptionName: testk8ssub
      # Required
      connection: sb-connection-string
      # Optional
      queueLength: "30" # default 5

but still not work
And I get a strange behaviour because the HPA is looking for a CPU metrics

k -n hpa describe hpa keda-hpa-consumer
Name:                                                  keda-hpa-consumer
Namespace:                                             hpa
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Tue, 07 May 2019 00:02:10 +0200
Reference:                                             Deployment/consumer
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 80%
Min replicas:                                          1
Max replicas:                                          10
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedComputeMetricsReplicas  1m (x12 over 4m)  horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu
  Warning  FailedGetResourceMetric       1m (x13 over 4m)  horizontal-pod-autoscaler  missing request for cpu

[aminebizid]

Could you please send me what do you obtain as hpa

k edit hpa

[lee0c]

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2019-05-06T23:22:41Z","reason":"SucceededGetScale","message":"the
      HPA controller was able to get the target''s current scale"},{"type":"ScalingActive","status":"False","lastTransitionTime":"2019-05-06T23:22:41Z","reason":"ScalingDisabled","message":"scaling
      is disabled since the replica count of the target is zero"}]'
    autoscaling.alpha.kubernetes.io/metrics: '[{"type":"External","external":{"metricName":"queueLength","metricSelector":{"matchLabels":{"deploymentName":"lecattar-t"}},"targetAverageValue":"5"}}]'
  creationTimestamp: 2019-05-06T23:22:26Z
  name: keda-hpa-lecattar-t
  namespace: default
  resourceVersion: "2073022"
  selfLink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/keda-hpa-lecattar-t
  uid: cf513b75-7055-11e9-85aa-2a21f218ad39
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: lecattar-t
status:
  currentReplicas: 0
  desiredReplicas: 0

The CPU scaling instead of using an external metric will happen if external metrics aren't successfully added to the HPA - likely that there's still an issue with the connection string or you're otherwise missing something necessary to create the external metric spec that gets attached to the HPA.

I did some digging & think I found a bug that is causing this. I'm going to look into it more & get back to you.

[lee0c]

The linked PR should sort this out, looks like there was a bug in how environment variables set with KeyRefs were resolved.

[aminebizid]

Now I get this issue:

time="2019-05-07T07:25:54Z" level=error msg="Error updating scaledObject (hp/azure-servicebus-queue-scaledobject) status: Operation cannot be fulfilled on scaledobjects.keda.k8s.io \"azure-servicebus-queue-scaledobject\": the object has been modified; please apply your changes to the latest version and try again"

E0507 07:29:09.908032       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"Exactly one scaled object should match label deploymentName=consumer"}

k -n hp get hpa
NAME                REFERENCE             TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-consumer   Deployment/consumer   <unknown>/30 (avg)   1         10        1          6m

Generated hpa

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2019-05-07T07:26:09Z","reason":"SucceededGetScale","message":"the
      HPA controller was able to get the target''s current scale"},{"type":"ScalingActive","status":"False","lastTransitionTime":"2019-05-07T07:26:09Z","reason":"FailedGetExternalMetric","message":"the
      HPA was unable to compute the replica count: unable to get external metric hp/queueLength/\u0026LabelSelector{MatchLabels:map[string]string{deploymentName:
      consumer,},MatchExpressions:[],}: unable to fetch metrics from external metrics
      API: Exactly one scaled object should match label deploymentName=consumer"}]'
    autoscaling.alpha.kubernetes.io/metrics: '[{"type":"External","external":{"metricName":"queueLength","metricSelector":{"matchLabels":{"deploymentName":"consumer"}},"targetAverageValue":"30"}}]'
  creationTimestamp: 2019-05-07T07:25:54Z
  name: keda-hpa-consumer
  namespace: hp
  resourceVersion: "3929482"
  selfLink: /apis/autoscaling/v1/namespaces/hp/horizontalpodautoscalers/keda-hpa-consumer
  uid: 595b796b-7099-11e9-8c5c-000d3a441aef
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: consumer
status:
  currentReplicas: 1
  desiredReplicas: 0

[aminebizid]

@lee0c Could you push a complete sample for Scaling on Azure Service Bus topic
ScaledObject and deployment yaml files

[lee0c]

@aminebizid Glad you opened a new issue for the new problem. My deployment used Azure Functions so it likely won't be super useful to you unfortunately.