Kube-prometheus: RUNTIME ERROR: Can only base64 encode strings / arrays of single bytes.

[DimitrijeManic]

What happened?
When adding a slack receiver to the example found in alertmanager-configuration. The build command fails with the error

RUNTIME ERROR: Can only base64 encode strings / arrays of single bytes.

alertmanager-config.yaml

global:
  resolve_timeout: 10m
route:
  group_by: ["job", "alertname"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: "null"
  routes:
    - match:
        severity: critical
      receiver: "slack"
    - match:
        severity: warning
      receiver: "slack"
  receivers:
    - name: "slack"
      slack_configs:
        - send_resolved: true
          username: "Prometheus"
          channel: "#channel_name"
          api_url: "https://hooks.slack.com/services/XXXX/XXXXX/XXXXXXX"
          title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Monitoring Event Notification'
          text: >-
            {{ range .Alerts }}
              *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
              *Description:* {{ .Annotations.message }}
              *Logs:* <https://xxxxxx.com/logs/view?tags=test,k8s&hosts={{ .Labels.pod }}|:blue_book:>
              *Runbook:* <https://xxxxx.com/master/runbook.md#{{ .Labels.alertname }}|:spiral_note_pad:>
              *Details:*
              {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
              {{ end }}
            {{ end }}
    - name: "null"

I Believe that the error comes from jsonnet not being able to parse the nested "slack_configs" object?

I'm curious how everyone has gotten around this?

Did you expect to see some different?
I was expecting the build to parse successfully

How to reproduce it (as minimally and precisely as possible):
Run the alertmanager-configuration example and add the following slack receiver

  receivers:
    - name: "slack"
      slack_configs:
        - send_resolved: true
          username: "Prometheus"
          channel: "@channel_name"
          api_url: "https://hooks.slack.com/services/XXXX/XXXXX/XXXXXXX"
          title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Monitoring Event Notification'
          text: >-
            {{ range .Alerts }}
              *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
              *Description:* {{ .Annotations.message }}
              *Logs:* <https://xxxxxx.com/logs/view?tags=test,k8s&hosts={{ .Labels.pod }}|:blue_book:>
              *Runbook:* <https://xxxxx.com/master/runbook.md#{{ .Labels.alertname }}|:spiral_note_pad:>
              *Details:*
              {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
              {{ end }}
            {{ end }}

[metalmatze]

Could you share the entire file you use to generate this?
The important part of the files is that config: ||| starts a multi-line string in jsonnet and the contents of that string are arbitrary for jsonnet itself.

[DimitrijeManic]

@metalmatze thanks for replying,

alertmanager-config.yaml

global:
  resolve_timeout: 10m
route:
  group_by: ["job", "alertname"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: "null"
  routes:
    - match:
        severity: critical
      receiver: "slack"
    - match:
        severity: warning
      receiver: "slack"
  receivers:
    - name: "slack"
      slack_configs:
        - send_resolved: true
          username: "Prometheus"
          channel: "#channel_name"
          api_url: "https://hooks.slack.com/services/XXXX/XXXXX/XXXXXXX"
          title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Monitoring Event Notification'
          text: >-
            {{ range .Alerts }}
              *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
              *Description:* {{ .Annotations.message }}
              *Logs:* <https://xxxxxx.com/logs/view?tags=test,k8s&hosts={{ .Labels.pod }}|:blue_book:>
              *Runbook:* <https://xxxxx.com/master/runbook.md#{{ .Labels.alertname }}|:spiral_note_pad:>
              *Details:*
              {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
              {{ end }}
            {{ end }}
    - name: "null"

main.jsonnet

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
      alertmanager+:: {
        config: importstr 'alertmanager-config.yaml',
      }
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) }
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) }
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

I have also tried in-line and I get the same error
main.jsonnet

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
      alertmanager+:: {
        config: |||
          global:
            resolve_timeout: 10m
          route:
            group_by: ["job", "alertname"]
            group_wait: 30s
            group_interval: 5m
            repeat_interval: 3h
            receiver: "null"
            routes:
              - match:
                  severity: critical
                receiver: "slack"
              - match:
                  severity: warning
                receiver: "slack"
            receivers:
              - name: "slack"
                slack_configs:
                  - send_resolved: true
                    username: "Prometheus"
                    channel: "#channel_name"
                    api_url: "https://hooks.slack.com/services/XXXX/XXXXX/XXXXXXX"
                    title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Monitoring Event Notification'
                    text: >-
                      {{ range .Alerts }}
                        *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
                        *Description:* {{ .Annotations.message }}
                        *Logs:* <https://xxxxxx.com/logs/view?tags=test,k8s&hosts={{ .Labels.pod }}|:blue_book:>
                        *Runbook:* <https://xxxxx.com/master/runbook.md#{{ .Labels.alertname }}|:spiral_note_pad:>
                        *Details:*
                        {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
                        {{ end }}
                      {{ end }}
              - name: "null"
        |||,
      }
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) }
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) }
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

The command used to build:

jsonnet -J vendor -m manifests "${1-main.jsonnet}" | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml; rm -f {}' -- {}

[DimitrijeManic]

The problem came from the special character in the slack message template

{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`

Once I removed the • everything worked as expected.

Related ticket: https://github.com/google/jsonnet/issues/575