Keda: Azure Storage Queue Scaler Issue

[peishuli]

I wrote a Azure Storage Queue triggered function and it ran fine locally. Once deployed to my local k8s cluster (Docker Desktop on win10), a pod was created but was terminated right away. No pods would ever been created even I pushed tons of messages to the queue.

I checked the keda-operator pod and found the following error:

{
  "level": "error",
  "ts": 1589679556.2457619,
  "logger": "azure_queue_scaler",
  "msg": "error)",
  "**error**": "**Can't parse storage connection string**",
  "stacktrace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/kedacore/keda/pkg/scalers.(*azureQueueScaler).IsActive\n\tkeda/pkg/scalers/azure_queue_scaler.go:123\ngithub.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScaleDeployment\n\tkeda/pkg/handler/scale_loop.go:119\ngithub.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScale\n\tkeda/pkg/handler/scale_loop.go:45\ngithub.com/kedacore/keda/pkg/handler.(*ScaleHandler).HandleScaleLoop\n\tkeda/pkg/handler/scale_loop.go:28"
}

Below is the k8s menifests yaml file:

data:
  AzureWebJobsStorage: <based64encodedstorageaccoutconnectionstring>
  FUNCTIONS_WORKER_RUNTIME: ZG90bmV0
apiVersion: v1
kind: Secret
metadata:
  name: functionapp1
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: functionapp1
  namespace: default
  labels:
    app: functionapp1
spec:
  selector:
    matchLabels:
      app: functionapp1
  template:
    metadata:
      labels:
        app: functionapp1
    spec:
      containers:
      - name: functionapp1
        image: peishu/functionapp1
        env:
        - name: AzureFunctionsJobHost__functions__0
          value: Function1
        envFrom:
        - secretRef:
            name: functionapp1
---
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: functionapp1
  namespace: default
  labels:
    deploymentName: functionapp1
spec:
  scaleTargetRef:
    deploymentName: functionapp1
  pollingInterval: 10
  cooldownPeriod:  30
  triggers:
  - type: azure-queue
    metadata:
      type: queueTrigger
      connection: AzureWebJobsStorage
      queueName: kedademo
      name: myQueueItem
      queueLength: '2'
---

More informaiton:
If I add _minReplicaCount: 1_ in the ScaledObject spec, and apply the change, it will created a pod but won't scale up pod replicas even there are thousands of messages in the queue and the keda-operator pod's log will be full of the "Can't parse storage connection string" errors shown above.

Thanks for the help!!

[peishuli]

I reviewed the scaler source code and figured it out the cause of the issue.

Below is the code that does the parsing:

func ParseAzureStorageConnectionString(connectionString string) (string, string, string, string, error) {
    parts := strings.Split(connectionString, ";")

    var endpointProtocol, name, key, endpointSuffix string
    for _, v := range parts {
        if strings.HasPrefix(v, "DefaultEndpointsProtocol") {
            protocolParts := strings.SplitN(v, "=", 2)
            if len(protocolParts) == 2 {
                endpointProtocol = protocolParts[1]
            }
        } else if strings.HasPrefix(v, "AccountName") {
            accountParts := strings.SplitN(v, "=", 2)
            if len(accountParts) == 2 {
                name = accountParts[1]
            }
        } else if strings.HasPrefix(v, "AccountKey") {
            keyParts := strings.SplitN(v, "=", 2)
            if len(keyParts) == 2 {
                key = keyParts[1]
            }
        } else if strings.HasPrefix(v, "EndpointSuffix") {
            suffixParts := strings.SplitN(v, "=", 2)
            if len(suffixParts) == 2 {
                endpointSuffix = suffixParts[1]
            }
        }
    }
    if name == "" || key == "" || endpointProtocol == "" || endpointSuffix == "" {
        return "", "", "", "", errors.New("Can't parse storage connection string")
    }

    return endpointProtocol, name, key, endpointSuffix, nil
}

It expects a connectionstring that consists of four ";"-separted key/value pairs:

• DefaultEndpointsProtocol
• AccountName
• AccountKey
• EndpointSuffix

This is the format displayed in Azure Portal. The connection string I used was copied from the Properties window of Microsoft Azure Storage Explorer and it has slightly different key/values pairs:

• DefaultEndpointsProtocol
• AccountName
• AccountKey
• BlobEndpoint
• QueueEndpoint
• TableEndpoint
• FileEndpoint

This won't cause problems for my function code (written in C#) but would blow scaler's parsing routine.

I'm not sure it is a bug but I would certainly suggest enhance the _ParseAzureStorageConnectionString_ function (in AzureStorage.go) to accommodate both formats.

Thanks for your attention!

[ahmelsayed]

Thanks for the detailed investigation @peishuli I agree the parse method should handle this.