Azure-docs: Bug when deploying Application Insights custom metric alert

[johndowns]

There appears to be a bug when deploying metric alerts with custom Application Insights metrics.

If I deploy a metric alert using an ARM template, as documented on this page, I have to specify the custom metric name in the metricName property. However, when I deploy the template, I get the following deployment error:

{
  "Code": "BadRequest",
  "Message": "The metric names were not found MyCustomMetric."
}

I assume this is because there is no data with that metric name in Application Insights at the time of deployment - but I wouldn't expect there to be until my application runs and emits those metrics. In my view this is a bug, since I should be able to set up an alert without the custom metric having already been used. This would always be the case when I deploy a new environment, for example.

Here's an example ARM definition for a metric alert that illustrates this problem:

{
    "name": "MyCustomMetricAlert",
    "type": "Microsoft.Insights/metricAlerts",
    "location": "global",
    "apiVersion": "2018-03-01",
    "properties": {
        "description": "My description",
        "enabled": true,
        "severity": 2,
        "windowSize": "PT1H",
        "evaluationFrequency": "PT1H",
        "scopes": [
            "[resourceId('Microsoft.Insights/components', variables('appInsightsName'))]"
        ],
        "criteria": {
            "allOf": [
                {
                    "name": "MyCustomMetric less than or equal to zero",
                    "metricName": "MyCustomMetric",
                    "metricNamespace": "Azure.ApplicationInsights",
                    "operator": "GreaterThan",
                    "threshold": 0,
                    "timeAggregation": "Total"
                }
            ],
            "odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria"
        },
        "actions": [
            {
                "actionGroupId": "[resourceId('Microsoft.Insights/actionGroups', variables('alertsActionGroupName'))]",
                "webhookProperties": {}
            }
        ],
        "autoMitigate": true
    },
    "dependsOn": [
        "[resourceId('Microsoft.Insights/actionGroups', variables('alertsActionGroupName'))]",
        "[resourceId('Microsoft.Insights/components', variables('appInsightsName'))]"
    ]
}

I have prepared a complete ARM template that illustrates the problem here as well.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 1f2bd9d0-af11-d684-051b-7ac8e611cca9
• Version Independent ID: f83d6e85-4d52-6b7f-a79e-f3ed3f3d0a95
• Content: Create a metric alert with a Resource Manager template
• Content Source: articles/monitoring-and-diagnostics/monitoring-create-metric-alerts-with-templates.md
• Service: azure-monitor
• GitHub Login: @snehithm
• Microsoft Alias: snmuvva

[Karishma-Tiwari-MSFT]

Thanks for the feedback! We are currently investigating and will update you shortly.

[femsulu]

@snehithm - any insights on this please?

[femsulu]

assigning to content owner for deeper review.

[snehithm]

Hi @johndowns,

Thanks for reporting it. I will investigate and get back to you. Could you email me at snmuvva at microsoft dot com? I would like to reach out as I need some details about how you are emitting custom metrics today?

Regards,
Snehith

[johndowns]

Hi @snehithm, I've emailed you as requested.

[ruprizal]

Hello @snehithm, any updates on this? I am getting the same bad request error when I try to deploy Metricalert for my Streaming Analytics job.
Please provide feedback..

[ruprizal]

@johndowns - did you figure out a way to fix this deployment issue? I am facing the same issue now.

[johndowns]

@ruprizal I've been in email contact with @snehithm and colleagues, and they've acknowledged this isn't the behaviour we might expect. I haven't heard of any plans for a fix yet but I followed up earlier this week.

[ruprizal]

I found the issue. The metricName should be one of the predefined supported metric names from MS.
In the example above, "metricName": "MyCustomMetric", is invalid. Example: if you are tracking run time errors, then example metricName should be like this: "metricName":"Errors"

Please use one of the supported metricNames from the link below:
https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-supported-metrics

[ruprizal]

Here is specific link to supported stream analytics job metric names:
https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-supported-metrics#microsoftstreamanalyticsstreamingjobs

[johndowns]

@ruprizal I'm glad you fixed your problem, but this is not the point of this issue. This issue is specifically about custom metric alerts in Application Insights. Application Insights allows for creating your own metrics and then alerting based on them, so there is no predefined list. (Additionally Azure Monitor now supports custom metrics on all resource types too, so the issue will occur with these too.)

Furthermore as I mentioned above, the product team has acknowledged this bug, but I haven't heard anything since then with regards to a fix.

[debbiemcg]

We have encountered the same issue. Furthermore, when we deploy an update to a stopped Azure function (V1), if the function has not been running for some period of time, the existing metrics disappear and cannot be referenced again until the functions are started and some activity is recorded. This is fine for TEST and QA environments, but not being able to set alerts on these metrics when the function first is deployed and started in Production seems like a serious oversight.

[eaidland]

If you deploy an arm template with metric alerts, then disable them in the portal and redeploy you'll get an internal server error.

[PaulSells]

When deploying a metric alert using the template there appears to be an issue when setting the action group. I'm passing in the the full resource id of an existing ActionGroup in the format

/subscriptions//providers/microsoft.insights/actionGroups/

I've also tried "[resourceId('microsoft.insights/actionGroups', parameters('actionGroupId'))]"

The metric alert is created successfully in both instances however the action group on the alert is empty?

[PaulSells]

Noticed a typo in my previous post resource id for action group should read:

/subscriptions//resourceGroups//providers/microsoft.insights/actionGroups/

[razvansalajan]

Any updates ?

[chayanbaner]

Hi,
below gist by me might help. Here I am creating a ping test with metric alert

https://gist.github.com/chayanbaner/f8f8234638bc07d3cfe6327073492575

[hoangpx]

I got this message "The metric names were not found QueueMessageCount". I'm using resourceID: /subscriptions//resourceGroups//providers/Microsoft.Storage/storageAccounts/

[romantitov]

Any updates how to fix initial issue? Any workarounds?
Currently we have to make a custom metrics for our alerts manually before deployment. But even in this case when the custom metric didn't send to application insight for a few days next deployment will fail with the same error

{ "Code": "BadRequest", "Message": "The metric names were not found MyCustomMetric." }

[romantitov]

I've found another solution by using Microsoft.Insights/scheduledQueryRules for my alerts based on custom metrics, which is nor required my custom metric to be created before deployment

Examples:
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/alerts-log
https://docs.microsoft.com/en-us/rest/api/monitor/scheduledqueryrules/createorupdate

[psyduck101]

@snehithm Any updates from the product team on the initial issue? I have the same problem with a custom metric, that we need for the moment.

[felipecruz91]

Any updates?

[snehithm]

Apologies, I didn't realize I still had this. @harelbr who is current owner for this area.

[harelbr]

We have started the work for supporting this scenario, and will update once this is available.

[harelbr]

I'm happy to share that we've just added support for creating a metric alert rule on a custom metric that isn't yet emitted.

This can be done by passing a skipMetricValidation parameter (within the criteria) and setting it to "true".

See example below for how to use this parameter in an ARM template:

"criteria": {
       "odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria",
       "allOf": [
                        {
                            "name" : "1st criterion",
                            "metricName": "[parameters('metricName')]",
                        "metricNamespace": "[parameters('metricNamespace')]",
                            "dimensions":[],
                            "operator": "[parameters('operator')]",
                            "threshold" : "[parameters('threshold')]",
                            "timeAggregation": "[parameters('timeAggregation')]",
                        "skipMetricValidation": true
                        }
                    ]
                }

[Chryogenic]

@harelbr - I can't see this change reflected in the documentation:
https://docs.microsoft.com/en-us/azure/templates/microsoft.insights/metricalerts

Is "skipMetricValidation" working at the moment?

Thanks

[harelbr]

@Chryogenic - Thanks for reaching out!

Yes, skipMetricValidation is working, we're in the process of adding documentation for this parameter.

[jstevans]

@harelbr Is this available via the REST API, or will it be? We'd like to open a PR against the Terraform provider, but need upstream (REST spec => Azure Go SDK) to have support first.

[harelbr]

@jstevans - Yes, this is available in the metric alerts REST API (although you currently can't see it in the documentation).

Currently, this field doesn't appear in the swagger, so SDKs generated from it (e.g. the Azure Go SDK) won't get this parameter. We will expose this parameter in the swagger in future versions (no ETA yet). Until then, you can use this parameter by directly calling the REST API.

Thanks!

[harelbr]

please-close