Victoriametrics: VMAlert: {{$labels.alertname}} doesn't work in rules config

[BenB196]

Describe the bug
The variable {{$labels.alertname}} doesn't work within VMAlert, but does work with Prometheus alerts

To Reproduce
Have alert rule like:

groups:
- name: Cisco Router Rules
  rules:
    - alert: Router IP SLA
      expr: rttMonLatestRttOperSense{type=~".*router"} != 1
      for: 10m
      labels:
        severity: critical
        instance: '{{ $labels.instance }}'
        event: 'Router IP SLA {{ $labels.rttMonCtrlAdminIndex }} Down'
        alertname: '{{ $labels.alertname }}'
      annotations:
        value: '{{ $value }}'
        description: "Router: {{ $labels.instance }} has IP SLA {{ $labels.rttMonCtrlAdminIndex }} down for greater than 10 minutes"

Expected behavior
Expect that labels.alertname to be populated with the alert name of "Router IP SLA"

Instead labels.alertname is ""

      {
        "id": "10733165428505242596",
        "name": "Router IP SLA",
        "group_id": "10993062752098873792",
        "expression": "rttMonLatestRttOperSense{type=~\".*router\"} != 1",
        "state": "firing",
        "value": "4e+00",
        "labels": {
          "alertname": "",
          "event": "Router IP SLA 21 Down",
          "instance": "router01",
          "job": "snmp_router",
          "rttMonCtrlAdminIndex": "21",
          "severity": "critical",
          "type": "router"
        },
        "annotations": {
          "description": "Router: router01 has IP SLA 21 down for greater than 10 minutes",
          "value": "4"
        },
        "activeAt": "2020-09-04T20:55:18.213758096Z"
      }

Version
Docker container:

/ # ./vmalert-prod -version
vmalert-20200903-224810-tags-v1.40.1-0-gbe6ae4b5e

Used command-line flags

flag{name="datasource.basicAuth.password", value="secret"} 1
flag{name="datasource.basicAuth.username", value="vicMetrics"} 1
flag{name="datasource.tlsCAFile", value=""} 1
flag{name="datasource.tlsCertFile", value=""} 1
flag{name="datasource.tlsInsecureSkipVerify", value="false"} 1
flag{name="datasource.tlsKeyFile", value="secret"} 1
flag{name="datasource.tlsServerName", value=""} 1
flag{name="datasource.url", value="http://victoria-metrics01:8428"} 1
flag{name="enableTCP6", value="false"} 1
flag{name="envflag.enable", value="false"} 1
flag{name="envflag.prefix", value=""} 1
flag{name="evaluationInterval", value="1m0s"} 1
flag{name="external.alert.source", value=""} 1
flag{name="external.label", value=""} 1
flag{name="external.url", value=""} 1
flag{name="http.connTimeout", value="2m0s"} 1
flag{name="http.disableResponseCompression", value="false"} 1
flag{name="http.idleConnTimeout", value="1m0s"} 1
flag{name="http.maxGracefulShutdownDuration", value="7s"} 1
flag{name="http.pathPrefix", value=""} 1
flag{name="http.shutdownDelay", value="0s"} 1
flag{name="httpAuth.password", value="secret"} 1
flag{name="httpAuth.username", value=""} 1
flag{name="httpListenAddr", value=":8880"} 1
flag{name="loggerErrorsPerSecondLimit", value="10"} 1
flag{name="loggerFormat", value="default"} 1
flag{name="loggerLevel", value="INFO"} 1
flag{name="loggerOutput", value="stderr"} 1
flag{name="memory.allowedBytes", value="0"} 1
flag{name="memory.allowedPercent", value="60"} 1
flag{name="metricsAuthKey", value="secret"} 1
flag{name="notifier.basicAuth.password", value="secret"} 1
flag{name="notifier.basicAuth.username", value=""} 1
flag{name="notifier.tlsCAFile", value=""} 1
flag{name="notifier.tlsCertFile", value=""} 1
flag{name="notifier.tlsInsecureSkipVerify", value="false"} 1
flag{name="notifier.tlsKeyFile", value="secret"} 1
flag{name="notifier.tlsServerName", value=""} 1
flag{name="notifier.url", value="http://alertmanager01:9093"} 1
flag{name="pprofAuthKey", value="secret"} 1
flag{name="remoteRead.basicAuth.password", value="secret"} 1
flag{name="remoteRead.basicAuth.username", value="vicMetrics"} 1
flag{name="remoteRead.lookback", value="1h0m0s"} 1
flag{name="remoteRead.tlsCAFile", value=""} 1
flag{name="remoteRead.tlsCertFile", value=""} 1
flag{name="remoteRead.tlsInsecureSkipVerify", value="false"} 1
flag{name="remoteRead.tlsKeyFile", value="secret"} 1
flag{name="remoteRead.tlsServerName", value=""} 1
flag{name="remoteRead.url", value="http://victoria-metrics01:8428"} 1
flag{name="remoteWrite.basicAuth.password", value="secret"} 1
flag{name="remoteWrite.basicAuth.username", value="vicMetrics"} 1
flag{name="remoteWrite.concurrency", value="1"} 1
flag{name="remoteWrite.flushInterval", value="5s"} 1
flag{name="remoteWrite.maxBatchSize", value="1000"} 1
flag{name="remoteWrite.maxQueueSize", value="100000"} 1
flag{name="remoteWrite.tlsCAFile", value=""} 1
flag{name="remoteWrite.tlsCertFile", value=""} 1
flag{name="remoteWrite.tlsInsecureSkipVerify", value="false"} 1
flag{name="remoteWrite.tlsKeyFile", value="secret"} 1
flag{name="remoteWrite.tlsServerName", value=""} 1
flag{name="remoteWrite.url", value="http://victoria-metrics01:8428"} 1
flag{name="rule", value="/etc/prometheus/rules/*.rules"} 1
flag{name="rule.validateExpressions", value="true"} 1
flag{name="rule.validateTemplates", value="true"} 1
flag{name="tls", value="false"} 1
flag{name="tlsCertFile", value=""} 1
flag{name="tlsKeyFile", value="secret"} 1
flag{name="version", value="false"} 1

[f41gh7]

In addition to alertname, prometheus provides label alertstate.

@hagen1778 WDYT?

[hagen1778]

Hi @BenB196 @f41gh7 ! Thanks for the issue!

@BenB196 alertname is automatically attached to every sent alert to the alertmanager, so there is no need to do it explicitly. I've added the same rule but with different env to Prometheus and VM :

alert: ExampleAlertAlwaysFiring
expr: sum by(job) (up == 1)
labels:
  env: prom
  subname: '{{$labels.alertname}}'
annotations:
  summary: '{{$labels.alertname}} ({{$labels.alertstate}})'

Generated alert in alertmanager for Prometheus (2.21.0) looks like following:

Generated alert in alertmanager for VM looks like following:

Do you observe different behaviour?

@f41gh7 alertstate label is set for generated time series in both VM and Prometheus:

ALERTS{alertname="ExampleAlertAlwaysFiring", alertstate="firing", env="vm", job="vmagent", subname="ExampleAlertAlwaysFiring"}
ALERTS{alertname="ExampleAlertAlwaysFiring", alertstate="firing", env="vm", job="victoriametrics", subname="ExampleAlertAlwaysFiring"}

[BenB196]

@hagen1778 thanks for the clarification, omitting the label entirely, allows it to be passed without the need for an additional label.

[hagen1778]

Closing as solved