Keda: [Scaler] Azure Monitor

[jeffhollan]

A generic Azure Monitor trigger for resources that support monitor but don’t yet have a scaler.

/cc @jsturtevant

[tomkerkhove]

Bumping this as this came up today by @jornbeyers.

Having this scaler would allow customers to target any Azure resource without having a dedicated scaler. This would end up being a "generic Azure scaler" where the scalers like Azure Service Bus are more scalers that make it more convenient.

There are two flavors of this that I see:

• Query metric in Monitor based on customer input (similar to how Promitor does this)
  
  
  
  • Required info is metricName, resourceUri, resourceGroup, subscriptionId
  
  
• Run in-line Kusto query (advanced)

Happy to flesh out a spec proposal if you want

[tomkerkhove]

Adding this to v1.1 to support metrics we don't expose via our other Azure scalers yet.

Specing out next week.

[melmaliacone]

@tomkerkhove, per my manager's suggestion I was thinking of picking this up. Do you have some time this week to sync on this? And I can align my code with your spec.

[tomkerkhove]

I'm going out-of-office tomorrow so might be a bit short notice, but I had something like this in mind.

Authentication

My suggestion would be to support managed identity & service principle-based auth.
Our docs should also clearly document what permissions should be granted, in this case Monitor Reader should be sufficient, similar to Promitor

Managed Identity

MI is super simple:

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-monitor
spec:
  podIdentity:
    provider: azure

Service Principle

I'd suggest to use following parameters for Service Principle-based auth, but they are just suggestions:

• adServicePrincipleId - Application ID of SP
• adServicePrinciplePassword - Password of SP

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-monitor
spec:
  podIdentity:
      provider: azure
  env:
  - parameter: adServicePrincipleId
    name: MONITOR_AD_APPID
  secretTargetRef:
  - parameter: adServicePrinciplePassword
    name: secrets-demo
    key: ad-sp-password

Trigger Metadata

There are a couple of options:

• Pragmatic - Specify subscriptionId & resourceGroupName as seperate entries
• Advanced - Enforce full uri a la subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ServiceBus/namespaces/contoso-orders

Personally I'm more of a fan of the pragmatic approach as people just have to provide the info and we compose the full resource URL which makes it easier to use it given they don't have to know/care how it works

Pragmatic

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: order-processor-scaler
  labels:
    app: order-processor
    deploymentName: order-processor
spec:
  scaleTargetRef:
    deploymentName: order-processor
  # minReplicaCount: 0 Change to define how many minimum replicas you want
  maxReplicaCount: 10
  triggers:
  - type: azure-monitor
    metadata:
      resource:
        uri: Microsoft.ServiceBus/namespaces/contoso-orders # Required. URI to the Azure resource
        tenantId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # Required. Used for authentication (unless I'm mistaken)
        subscriptionId: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy # Required. Used for determining the full resource URI
        resourceGroupName: keda # Required. Used for determining the full resource URI
      metric:
        filter: EntityName eq 'orders' # Optional. Used to define more specific part of the resource, in this example a queue in SB namespace
        name: ActiveMessages # Required. Officially supported metric from https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported
        aggregation:
          interval: 00:15:00 # Optional. Interval over which the metric values should be aggregated and reported
          type: Total # Required. Aggregation type to use for metric based on what is supported https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported
      auth:
        adServicePrincipleId: ABC # Optional. Used when no trigger auth is linked
        adServicePrinciplePassword: xxx # Optional. Used when no trigger auth is linked
    authenticationRef:
      name: trigger-auth-monitor

Advanced

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: order-processor-scaler
  labels:
    app: order-processor
    deploymentName: order-processor
spec:
  scaleTargetRef:
    deploymentName: order-processor
  # minReplicaCount: 0 Change to define how many minimum replicas you want
  maxReplicaCount: 10
  triggers:
  - type: azure-monitor
    metadata:
      resource:
        uri: subscriptions/xxx/resourceGroups/yyy/providers/Microsoft.ServiceBus/namespaces/contoso-ordersMicrosoft.ServiceBus/namespaces/contoso-orders # Required. URI to the Azure resource
        tenantId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # Required. Used for authentication (unless I'm mistaken)
      metric:
        filter: EntityName eq 'orders' # Optional. Used to define more specific part of the resource, in this example a queue in SB namespace
        name: ActiveMessages # Required. Officially supported metric from https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported
        aggregation:
          interval: 00:15:00 # Optional. Interval over which the metric values should be aggregated and reported
          type: Total # Required. Aggregation type to use for metric based on what is supported https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported
      auth:
        adServicePrincipleId: ABC # Optional. Used when no trigger auth is linked
        adServicePrinciplePassword: xxx # Optional. Used when no trigger auth is linked
    authenticationRef:
      name: trigger-auth-monitor

Closing

I've based this proposal of what I've seen on Promitor and is very similar to what we provide with our Generic Metric Scraper.

From what I'm seeing one thing is missing which is how often our metric server should query Azure Monitor. I'd advise to use every one/five minutes since there is already a 5 min delay on Monitor which could otherwise result in incorrect metrics. This could be surfaced & tweaked in the spec via a property like scrape.interval or query.interval

What are your thoughts @jeffhollan @ahmelsayed @melmaliacone?

[melmaliacone]

@tomkerkhove This looks good to me, but I don't have as much context as everyone else on this thread. I agree with you on the pragmatic approach over the advanced. And one-five minutes sounds good to me as well. I'm going to wait for the others to respond and then pick this up Friday and start playing around.

[tomkerkhove]

Awesome thanks! Feel free to join the standup tomorrow if you have time, I'll try to join as well

[melmaliacone]

I'll make sure to be there! Thanks for mentioning it, Tom.

[tomkerkhove]

My pleasure! You can find it here https://github.com/kedacore/keda#community

[dmeytin]

This feature will allow to use KEDA with security sensitive workloads.
Instead of providing KEDA read access to the data plane, where messages with sensitive information is located, this feature will allow to grant KEDA access to metrics only.
I'm looking forward to start using it.

[tomkerkhove]

Sounds like a good scenario! Happy to say @melmaliacone is already working on this!

Any update on the progress @melmaliacone?

[melmaliacone]

@tomkerkhove, got caught up in work stuff but I'm heads down on this until I finish this. I expect to have a PR the end of this week or the beginning of next. I know the pieces I need to glue together, just need to actually do it.

[tomkerkhove]

Awesome, thank you!

[tomkerkhove]

Reopening for docs.

Would you mind sending a PR for that @melmaliacone?

[melmaliacone]

Reopening for docs.

Would you mind sending a PR for that @melmaliacone?

Will do! Ran into some computer issues that required a fresh OS install, but it's at the top of my list.

[tomkerkhove]

Thanks!

[tomkerkhove]

Done via https://github.com/kedacore/keda-docs/pull/82, thanks!