Helm-charts: [prometheus-kube-stack] Alertmanager seems impossible to configure

[celestialorb]

Describe the bug
After trying a few different combinations of setting Helm values I cannot seem to fully configure the Alertmanager.

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.3.0", GitCommit:"8a4aeec08d67a7b84472007529e8097ec3742105", GitTreeState:"dirty", GoVersion:"go1.14.7"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Which chart: kube-prometheus-stack

Which version of the chart: 12.0.4

What happened: I have been trying to deploy a custom configuration to the Alertmanager using the chart. I've tried various approaches, but none seem to work the way I expect. I started by using the alertmanager.config value and was able to get some values merged under global but couldn't specify receivers or route.

From there I tried using various combinations of values under alertmanager.alertmanagerSpec like useExistingSecret with configSecret or changing the alertmanagerConfigSelector with a custom ConfigMap. None of these seemed to have any effect on the configuration of the alertmanager.

I've been trying to use very minimal configurations, primarily just changing the global.resolve_timeout to try and find a combination that works.

What you expected to happen: I expect the configuration to change when using some of these combinations of Helm values to configure the alertmanager.

How to reproduce it (as minimally and precisely as possible):
Changed values of values.yaml (only put values which differ from the defaults):

values.yaml

key: value

The helm command that you execute and failing/misfunctioning:

For example:

helm install my-release prometheus-community/name-of-chart --version version --values values.yaml

Helm values set after installation/upgrade:

helm get values my-release

Anything else we need to know: I suspect I'm just misunderstanding and thus misusing these Helm values, what's the best way to just override the alertmanager configuration entirely using the Helm values?

[mashun4ek]

Hi @celestialorb! I also faced issues with configuring alertmanager, and still working on it. Although I can't help with ConfigSelectors or secrets, maybe try to use this simple snippet for your values.yaml and see if it's working.

config:
    global:
      resolve_timeout: 5m
      slack_api_url: ${your_slack_hook}
    route:
      group_by: ['job']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'null'
      routes:
      - match:
          alertname: Watchdog
        receiver: 'null'
      - receiver: 'slack-notifications'
        continue: true
    receivers:
    - name: 'null'
    - name: 'slack-notifications'
      slack_configs:
      - channel: ${your_slack_channel}
        title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Test Alertmanager'
        text: >-
          {{ range .Alerts }}
            *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
          {{ end }}
        send_resolved: true

[celestialorb]

@mashun4ek I added my slack receiver as the second receiver (after null) and that seems to work! I guess I can't override that null receiver in the first spot. Thanks!

[dpolicastro]

@mashun4ek I added my slack receiver as the second receiver (after null) and that seems to work! I guess I can't override that null receiver in the first spot. Thanks!

The first receiver is the default one, you should have been able to override it. For me it worked fine 🤔

[mashun4ek]

@celestialorb As @dpolicastro mentioned, you can set your main receiver in route section. Hope it will work for you 🙂

config:
    global:
      ...
    route:
      ...
      receiver: 'default-receiver'
      routes:
        ...
    receivers:
    - name: 'default-receiver'
      ...

[ciaranmcveigh5]

I've also had issues configuring the alertmanager, although you can use the alertmanager.config in the values.yaml and get it working I haven't been able to get the alertmanager.alertmanagerSpec.configSecret working which is needed if you're going to have access tokens in your config and want to keep them secret.

from the description it sounds like a change to this value should change the mounted secret to the alertmanager stateful set which is mounted at /etc/alertmanager/config/alertmanager.yaml.

it worked on some simple configs ie when i used this file https://github.com/marcel-dempers/docker-development-youtube-series/blob/master/monitoring/prometheus/kubernetes/1.18.4/alertmanager/alertmanager-secret.yaml

however doesn't appear to work when i included additional fields like slack_configs: etc. heres a sample of what i put in the secret

global:
  resolve_timeout: 5m
  slack_api_url: "xxx"
route:
  group_by: ['job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  #receiver: 'slack'
  routes:
  - match:
      alertname: DeadMansSwitch
    receiver: 'null'
  - match:
      alertname: xxx
    receiver: 'slack'
    continue: true
receivers:
- name: 'null'
- name: 'slack'
  slack_configs:
  - channel: '#xxx'
    send_resolved: false
    title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Monitoring Event Notification'
    text: >-
      {{ range .Alerts }}
        *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
        *Description:* {{ .Annotations.description }}
        *Graph:* <{{ .GeneratorURL }}|:chart_with_upwards_trend:> *Runbook:* <{{ .Annotations.runbook }}|:spiral_note_pad:>
        *Details:*
        {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
        {{ end }}
      {{ end }}

can see that theres a alertmanager-prometheus-stack-kube-prom-alertmanager-generated secret which appears to be copying my alertmanager.alertmanagerSpec.configSecret and mounting that into the stateful set but with the above config the alertmanager-prometheus-stack-kube-prom-alertmanager-generated seems to go back to a default value

route:
  receiver: 'null'
receivers:
- name: 'null'

any ideas on this, i know the alertmanager.alertmanagerSpec.configSecret is designed specifically to handle secret access tokens in config it's just that as far as i can see it doesn't work with the above config

Thanks

k8s version 1.18.9
helm version 3.4.1

[ekeih]

TL;DR: Please read the updates at the bottom of the post.

I think I can confirm the observations of @ciaranmcveigh5.

For me the helm chart creates the following secret and keeps it up to date when I apply any changes via my values.yaml:

kubectl get secrets alertmanager-kube-prometheus-stack-alertmanager -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d

global:
  resolve_timeout: 5m
receivers:
- name: "null"
- name: rocketchat
  webhook_configs:
  - send_resolved: true
    url: https://rocketchat.../hooks/.../...
- name: signalilo
  webhook_configs:
  - http_config:
      bearer_token: ...
    send_resolved: true
    url: https://.../webhook
route:
  group_by:
  - job
  group_interval: 5m
  group_wait: 30s
  receiver: rocketchat
  repeat_interval: 12h
  routes:
  - match_re:
      alertname: Watchdog|KubeAPILatencyHigh|CPUThrottlingHigh|AggregatedAPIErrors|AggregatedAPIDown|KubeAPIErrorBudgetBurn
    receiver: "null"
  - continue: true
    receiver: rocketchat
  - receiver: signalilo
    repeat_interval: 4m

But the operator never updates the existing ...-generated secret accordingly. It still contains an older version of the config.

kubectl get secrets alertmanager-kube-prometheus-stack-alertmanager-generated -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d

global:
  resolve_timeout: 5m
receivers:
- name: "null"
- name: rocketchat
  webhook_configs:
  - send_resolved: true
    url: https://rocketchat..../hooks/.../...
route:
  group_by:
  - job
  group_interval: 5m
  group_wait: 30s
  receiver: rocketchat
  repeat_interval: 12h
  routes:
  - match_re:
      alertname: Watchdog|KubeAPILatencyHigh|CPUThrottlingHigh|AggregatedAPIErrors|AggregatedAPIDown|KubeAPIErrorBudgetBurn
    receiver: "null"

I double-checked the config and the logs a hundred times, but I can't find any indication that something with the config is wrong. I increased the loglevel of the operator but it doesn't log any issues with those secrets. Looking at https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md#manually-managed-secret I am pretty sure it is supposed to use the secret (the alertmanager is called kube-prometheus-stack-alertmanager and I didn't set a custom value for configSecret).
Looking at the docs and the source I think it should work like this, but it does not and I am out of ideas what else to try. So any hints or additional infos would be greatly appreciated.

We are currently running 12.0.0 of the chart. Looking at the diff I don't think there are any related changes from the last ten days, but I can also try upgrading the chart.

Thanks in advance and please let me know if I can provide more information.

Update 1: Other things I checked:

• Neither the alertmanager nor prometheus are paused.
• I also restarted the operator a bunch of times.
• The apiserver does not log any weird things, restarted it anyway.
• Manually verified that the serviceaccount of the operator has access to the secret.

Update 2: It turned out that we had a low (100m) CPU limit on the prometheus-operator pod which completely broke the alertmanager and prometheus reconcile loops (the thanos reconcile loop stil works), not exactly sure in which way it breaks it, but the corresponding reconcile metrics never get increased with such limit. After raising the CPU limit everything is back to normal now.
So at least in our case this was not specific for the secret-sync but a general reconcile problem.

[questionmorc]

I am having the same issue.
No matter what i put in the values file it seems that alertmanager-kube-prometheus-stack-alertmanager-generated does not merge with my custom values.

alertmanager:
  config:
    global:
      resolve_timeout: 5m
      slack_api_url: blehbleh
    route:
      group_by: ['job']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'null'
      routes:
      - match:
          alertname: Watchdog
        receiver: 'null'
      - receiver: 'slack-notifications'
        continue: true
    receivers:
    - name: 'null'
    - name: 'slack-notifications'
      slack_configs:
      - channel: "#sys"
        title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Test Alertmanager'
        text: >-
          {{ range .Alerts }}
            *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
          {{ end }}
        send_resolved: true

Even weirder is that if my values file only contains whats above, the statefulset and pods for alertmanager are not even created.
The above is stored in a secret: alertmanager-kube-prometheus-stack-alertmanager

Only if i remove the config key from the yaml file does alertmanager create the statefulset and deployment.
Even if i explicitly enable alertmanger before the config.

Am I missing something glaringly obvious here?

EDIT:
After reading @ekeih comment a few times I modified the CPU and Memory limit and noticed I had some errors in the log and the operator was not able to unmarshal my config correctly.
So in short, make sure to check the operator logs for errors. My problem was that I was not sure where to check at first.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.