Keda: HPA (scaling) is not working with Azure function kafka trigger

[somak12]

I am using Keda installed in AKS and using Azure function as consumer to Kafka topic. Using the library azure-functions-kafka-extension for KafkaTrigger.

It seems like even under load the azure function is not scaling up. When I check the HPA logs it is giving error as below.

Expected Behavior

Under load the azure function should scale

Actual Behavior

Azure function with kafka trigger extension is not scaling up

Steps to Reproduce the Problem

1. Create azure function with Kafka trigger extension
2. Use Azure function core tool to install Keda onto AKS and deploy the function to AKS
3. Insert data into Kafka topic to test scalability

Specifications

• KEDA Version: docker.io/kedacore/keda:1.1.0
• Platform & Version: AKS
• Kubernetes Version: 1.14.8
• Scaler(s): Kafka

[tomkerkhove]

What do the KEDA logs say?

[ppatierno]

Yes the KEDA logs would be useful. Then, is the keda-metrics-adapter running fine? From the error message, it should be ok but it cannot get the lagThreshold metric exposed by the Kafka scaler. Anyway it's worth checking.
What's the ScaledObject as well?

[somak12]

Hi, Checked the keda-metrics-apiserver , it seems to run fine, please find the logs below.

I0207 14:17:08.673044       1 provider.go:46] keda_metrics_adapter/provider "msg"="starting"
I0207 14:17:08.673104       1 main.go:84] keda_metrics_adapter "msg"="starting adapter..."
I0207 14:17:08.952606       1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0207 14:17:09.758020       1 secure_serving.go:116] Serving securely on [::]:6443 

Below is the scaledobject:

Name:         aks-kafkatrigger-demo-func
Namespace:    default
Labels:       deploymentName=aks-kafkatrigger-demo-func
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"keda.k8s.io/v1alpha1","kind":"ScaledObject","metadata":{"annotations":{},"labels":{"deploymentName":"aks-kafkatrigger-demo-...
API Version:  keda.k8s.io/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2020-02-12T14:25:09Z
  Finalizers:
    finalizer.keda.k8s.io
  Generation:        4
  Resource Version:  2605037
  Self Link:         /apis/keda.k8s.io/v1alpha1/namespaces/default/scaledobjects/aks-kafkatrigger-demo-func
  UID:               78d90421-4da3-11ea-b621-eab5f7757c10
Spec:
  Scale Target Ref:
    Deployment Name:  aks-kafkatrigger-demo-func
  Triggers:
    Metadata:
      Authentication Mode:  notSet
      Broker List:    list of brokers
      Consumer Group:       aksfuncconsumergrp1
      Lag Threshold:        10
      Name:                 kafkaEvent
      Protocol:             notSet
      Topic:                my-example-topic
      Type:                 kafkaTrigger
    Type:                   kafka
Status:
  External Metric Names:
    lagThreshold
  Last Active Time:  2020-02-13T20:49:19Z
Events:              <none>

Let me grab the keda logs and get back ASAP.

[ppatierno]

why the broker list is empty in the scaled object? It has to be the bootstrap servers list for allowing the scaler to connect to the Apache Kafka cluster and getting information about the offset in order to compute the lag. The KEDA operator should have an error about missing the broker list.

[somak12]

No no. I just remove the broker list before posting the issue. Its there. Please find below the log from Keda on removing and redeploying the kubernetes yaml file.

{"level":"info","ts":1581628163.1119406,"logger":"controller_scaledobject","msg":"Reconciling ScaledObject","Request.Namespace":"default","Request.Name":"aks-kafkatrigger-demo-func"}
{"level":"info","ts":1581628163.1119971,"logger":"controller_scaledobject","msg":"Successfully finalized ScaledObject","Request.Namespace":"default","Request.Name":"aks-kafkatrigger-demo-func"}
{"level":"info","ts":1581628163.126022,"logger":"controller_scaledobject","msg":"Reconciling ScaledObject","Request.Namespace":"default","Request.Name":"aks-kafkatrigger-demo-func"}
{"level":"info","ts":1581628271.5533967,"logger":"controller_scaledobject","msg":"Reconciling ScaledObject","Request.Namespace":"default","Request.Name":"aks-kafkatrigger-demo-func"}
{"level":"info","ts":1581628271.5534472,"logger":"controller_scaledobject","msg":"Adding Finalizer for the ScaledObject","Request.Namespace":"default","Request.Name":"aks-kafkatrigger-demo-func"}
{"level":"info","ts":1581628271.5615633,"logger":"controller_scaledobject","msg":"Detecting ScaleType from ScaledObject","Request.Namespace":"default","Request.Name":"aks-kafkatrigger-demo-func"}
{"level":"info","ts":1581628271.5615816,"logger":"controller_scaledobject","msg":"Detected ScaleType = Deployment","Request.Namespace":"default","Request.Name":"aks-kafkatrigger-demo-func"}
{"level":"info","ts":1581628271.5615997,"logger":"controller_scaledobject","msg":"Creating a new HPA","Request.Namespace":"default","Request.Name":"aks-kafkatrigger-demo-func","HPA.Namespace":"default","HPA.Name":"keda-hpa-aks-kafkatrigger-demo-func"}

[ppatierno]

Are you sure that a Kafka client running in the same cluster is able to reach the Kafka brokers?

[somak12]

Yes the pod (azure function with kafkatrigger) is working fine and connecting to the brokers. Please find the logs below:

Now listening on: http://[::]:80
Application started. Press Ctrl+C to shut down.
info: Host.Triggers.Kafka[0]
      Assigned partitions: [my-example-topic [[0]], my-example-topic [[1]], my-example-topic [[2]], my-example-topic [[3]], my-example-topic [[4]], my-example-topic [[5]], my-example-topic [[6]], my-example-topic [[7]], my-example-topic [[8]], my-example-topic [[9]]]
info: Function.kafkatriggerdemo[0]
      Executing 'kafkatriggerdemo' (Reason='', Id=3a30bd19-c42f-4023-9e95-fa0867f74e6c)
info: Function.kafkatriggerdemo.User[0]
      C# Kafka trigger function processed: 1
info: Function.kafkatriggerdemo[0]
      Executed 'kafkatriggerdemo' (Succeeded, Id=3a30bd19-c42f-4023-9e95-fa0867f74e6c)
info: Host.Triggers.Kafka[0]
      Stored commit offset my-example-topic / [0] / 30655

This is log from the same pod which is not scaling under load. Sorry if I am doing some mistake from my end. I am pretty new to Keda and kafka.

[balchua]

@somak12 if you dont mind making the ScaledObject posted above code formatted? Its difficult to read. Im wondering if there is something missing.
Also just to confirm, your kafka doesn't have SASL enabled?

[zroubalik]

@somak12 could you please check APIService object, if it is correctly deployed and configured?

kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml

Setting log level to debug in the keda-operator deployment might help a little bit as well.

[somak12]

Hi @zroubalik it seems like this is not running in my cluster.

get apiservice v1beta1.external.metrics.k8s.io -o yaml
Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" not found

[somak12]

@somak12 if you dont mind making the ScaledObject posted above code formatted? Its difficult to read. Im wondering if there is something missing.
Also just to confirm, your kafka doesn't have SASL enabled?

Hi @balchua I have updated the scaledobject with proper formatting.
No SASL is not enabled in kafka.

[ppatierno]

So it seems you are missing this https://github.com/kedacore/keda/blob/master/deploy/24-metrics-api_service.yaml and more specifically for your 1.1.0 version is this one https://github.com/kedacore/keda/blob/v1.1.0/deploy/api_service.yaml
How did you deploy KEDA on the AKS cluster?

[somak12]

I used Azure function core tool to deploy Keda.

Installing Kubernetes scalers
This deploys KEDA and Osiris to your cluster which allows you to deploy your functions in a scale-to-zero by default.

func kubernetes install --namespace {namespace}

https://github.com/Azure/azure-functions-core-tools

[zroubalik]

OK, so this installation method might be broken since your setup is missing the APIService

@SatishRanjan @ahmelsayed PTAL

[tomkerkhove]

Opening https://github.com/Azure/azure-functions-core-tools/issues/1822 to track it over there as well.

[somak12]

Hi,
I installed the missing components manually as per above guideline and the scaling seems to work fine now.FYI. What I did is reinstalled Keda 1.1.0 from the source onto my AKS cluster. Hope the installation issue will get resolved from azure function core tool side. Thanks.

[tomkerkhove]

Let's hope that as well.

Since we have an issue open over there, is it ok for you to close this one?

[somak12]

Yes please.

[tomkerkhove]

Thanks!