Keda: Azure Monitor Scaler will panic if the timeseries has no data (Related to #679)

[Vishal2696]

This is a continuation of the issue #679 . The error is still there. It is yet to be fixed. Just now (25th March, 2020) faced this error. I setup a scaledobject that will trigger autoscaling based on the IncomingMessages metric of the eventhub. I encountered this error when I pointed to an event hub that has been inactive for quite amount of time. However this worked properly when I pointed to an event-hub that has active message flow.
Below is the error message I got

panic: runtime error: index out of range [0] with length 0

goroutine 815 [running]:
github.com/kedacore/keda/pkg/scalers.extractValue(0xc000808140, 0x10, 0xc0006baa50, 0x24, 0xc00011485e, 0x1a, 0xc000114840, 0x12, 0xc000114853, 0xa, ...)
        keda/pkg/scalers/azure_monitor.go:127 +0x7e1
github.com/kedacore/keda/pkg/scalers.getAzureMetric(0x218df20, 0xc0007e52b0, 0x2190ec0, 0xc000555cb0, 0x0, 0x0, 0xdf8475800, 0xd18c2e2800, 0x3, 0x6fc23ac00, ...)
        keda/pkg/scalers/azure_monitor.go:108 +0x4c5
github.com/kedacore/keda/pkg/scalers.executeRequest(0x218df20, 0xc0007e52b0, 0x2190ec0, 0xc000555cb0, 0x0, 0x0, 0xdf8475800, 0xd18c2e2800, 0x3, 0x6fc23ac00, ...)
        keda/pkg/scalers/azure_monitor.go:79 +0xab
github.com/kedacore/keda/pkg/scalers.GetAzureMetricValue(0x21d8640, 0xc0009ca940, 0xc0003f62c0, 0xc000652710, 0xc0007f5e00, 0x42de7f)
        keda/pkg/scalers/azure_monitor.go:40 +0xf9
github.com/kedacore/keda/pkg/scalers.(*azureMonitorScaler).IsActive(0xc0000c0be8, 0x21d8640, 0xc0009ca940, 0x1, 0x1, 0xc000a5b200)
        keda/pkg/scalers/azure_monitor_scaler.go:159 +0x46
github.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScaleDeployment(0xc0005558c0, 0x21d8640, 0xc0009ca940, 0xc0003ee000)
        keda/pkg/handler/scale_loop.go:119 +0x11a
github.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScale(0xc0005558c0, 0x21d8640, 0xc0009ca940, 0xc0003ee000)
        keda/pkg/handler/scale_loop.go:45 +0x6b
github.com/kedacore/keda/pkg/handler.(*ScaleHandler).HandleScaleLoop(0xc0005558c0, 0x21d8640, 0xc0009ca940, 0xc0003ee000)
        keda/pkg/handler/scale_loop.go:14 +0x1d2
created by github.com/kedacore/keda/pkg/controller/scaledobject.(*ReconcileScaledObject).startScaleLoop
        keda/pkg/controller/scaledobject/scaledobject_controller.go:330 +0x35f

Specifications

• KEDA Version: 1.3.0
• Kubernetes Version: 1.14.8
• Scaler(s): Azure-Monitor Scaler

[tomkerkhove]

Can you have a look at this @melmaliacone or are you busy with other work?

[Vishal2696]

@tomkerkhove I'll provide more details here. The error pops up whenever there is no metrics to report for quite a bit of time.(Refer screenshot below)

[tomkerkhove]

Thanks! Feels like we are not handling metrics without a value correctly so we should:

• Return -1 or so if API returns nothing and reflect that in metrics by using 0
• Return nil when things go wrong

[melmaliacone]

@tomkerkhove, I’ll look at this today.

[melmaliacone]

@tomkerkhove, @vishal8k this got fixed in issue #679 but has not made it into a release yet. @vishal8k are you using KEDA version 1.3.0 or building form source? you'll need to build from source to get the fix until there's a new release

[tomkerkhove]

Thanks for checking @melmaliacone!

[Vishal2696]

@melmaliacone I'm using 1.3.0. I've installed using helm. So how do I install from source? I hearing this for the 1st time. Can you point me to any reference link on how to deploy KEDA from source? That would be helpful.

[melmaliacone]

@vishal8k

Normally you could just install KEDA with the stable Helm chart, but in between releases, if there's a bug fix, the Helm chart won't be updated until a new release. This bug will be fixed in the next release, but I'm not sure when that will happen.

Until then, you have three options:

1. Pull the keda-metrics-adapter and keda images from my DockerHub. I built them from master today.
2. Build locally using Visual Studio Code Remote - Containers
3. Build locally directly on your machine

After following one of the above methods, you'll have the two docker images necessary to install the helm chart. If you're using a Kubernetes cluster in the cloud, you'll need to push these images to a container registry like DockerHub (unless you use my images).

Then you can deploy KEDA via Helm, replacing the image values in the Helm chart with the images you built. Detailed instructions for exactly how to do this can be found here.

Let me know if you still run into issues. Hopefully this will unblock you until a new release goes out.

[tomkerkhove]

Is this still pending? Think this is finished with 1.4, right?

[Vishal2696]

@tomkerkhove Yes the fix has been released with 1.4 but I didn't get time to test it personally. Anyway will close this. It should work.