Kube-prometheus: alertmanager-alertmanager.yaml is blank

[bitva77]

What happened?

Building kube-prometheus using this jsonnet file:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-eks.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-all-namespaces.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
      alertmanager+: {
        config: importstr 'alertmanager-config.yaml',
      },
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

everything get's outputted into the manifests directory fine but the alertmanager-alertmanager.yaml file is null. The secret yaml file is as I expect. just the yaml that actually deploys alertmanager is not generated.

Did you expect to see some different?

I did expect to be able to deploy alertmanager as well :)

How to reproduce it (as minimally and precisely as possible):

here's my build.sh:

#!/usr/bin/env bash

# This script uses arg $1 (name of *.jsonnet file to use) to generate the manifests/*.yaml files.

GOPATH=$HOME/gocode
PATH=$PATH:$HOME/gocode/bin:$HOME/go/bin
export PATH GOPATH

set -e
set -x
# only exit with zero if all commands of the pipeline exit successfully
set -o pipefail

# Make sure to use project tooling
PATH="$(pwd)/tmp/bin:${PATH}"

# Make sure to start with a clean 'manifests' dir
rm -rf manifests
mkdir -p manifests/setup

# optional, but we would like to generate yaml, not json
jsonnet -J vendor -m manifests "${1-config.jsonnet}" | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml; rm -f {}' -- {}

# Make sure to remove json files
find manifests -type f ! -name '*.yaml' -delete
rm -f kustomization

Environment

• Prometheus Operator version:

jb install github.com/coreos/kube-prometheus/jsonnet/[email protected]

Thanks!

[brancz]

Hmm interesting do the other files like the secret generate as expected? https://github.com/coreos/kube-prometheus/blob/2539ba9548dd108b59e10d9c7d2a2929804b3112/jsonnet/kube-prometheus/alertmanager/alertmanager.libsonnet#L76-L85

[bitva77]

Yes, the secret is created just fine.

service, serviceAccount and serviceMonitor are all created as well.

[brancz]

Strange .. I started an entirely new project with exactly your code and it works just fine for me. Is there anything else that you might be doing in your jsonnet code that you haven't shared yet?

[bitva77]

bizarre. just did it again from scratch and it's still blank. I'm copying the build.sh and config.jsonnet as above:

$ jb init

$ jb install github.com/coreos/kube-prometheus/jsonnet/[email protected]
GET https://github.com/coreos/kube-prometheus/archive/4e7440f742df31cd6da188f52ddc4e4037b81599.tar.gz 200
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/f0a5099c8214241842b827a08eba6d7b515550ea.tar.gz 200
GET https://github.com/brancz/kubernetes-grafana/archive/014301fd5f71d8305a395b2fb437089a7b1a3999.tar.gz 200
GET https://github.com/coreos/prometheus-operator/archive/8b9d024467383d84b55d7e5e0d4f7a33eb5007b3.tar.gz 200
GET https://github.com/coreos/etcd/archive/6c81b20ec8de0e3304708cf81a0713f1f0079083.tar.gz 200
GET https://github.com/prometheus/prometheus/archive/04b028f1e611a7887cbfc54585d965eb2cc5e7e6.tar.gz 200
GET https://github.com/prometheus/node_exporter/archive/503e4fc8486c0082d6bd8c53fad646bcfafeedf6.tar.gz 200
GET https://github.com/metalmatze/slo-libsonnet/archive/e238df4fac957357d78a405966c51523ef151cbc.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/f2a35172b97a0c944c4a167bb1f6e688624602e4.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/ad85aec356b4544a41f62ac8c32f8042c0ffc42e.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/6eab5fe2dfde77494c0452ce7ec3ed3ff21d9631.tar.gz 200

$ ./build.sh 
+ set -o pipefail
++ pwd
+ PATH='LONGASSPATH REDACTED'
+ rm -rf manifests
+ mkdir -p manifests/setup
+ jsonnet -J vendor -m manifests config.jsonnet
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml; rm -f {}' -- '{}'
cat: manifests/alertmanager-alertmanager: No such file or directory
+ find manifests -type f '!' -name '*.yaml' -delete
+ rm -f kustomization

cat: manifests/alertmanager-alertmanager: No such file or directory

weird....

I am doing this under WSL2 which might be the one weird thing but don't think that should have this type of an effect. I've previously built this project under WSL1...

[brancz]

hmm maybe let's disect this piece by piece then .. what does your manifests/ directory look like just after the jsonnet command?

jsonnet -J vendor -m manifests config.jsonnet

We've seen slight differences on OSX being freebsd coreutils by default instead of gnu, so I could see something similar being the case here.

[bitva77]

Cool, thanks.

So that generates this:

manifests/alertmanager-alertmanager
manifests/alertmanager-secret
manifests/alertmanager-service
manifests/alertmanager-serviceAccount
manifests/alertmanager-serviceMonitor
manifests/grafana-dashboardDatasources
manifests/grafana-dashboardDefinitions
manifests/grafana-dashboardSources
manifests/grafana-deployment
manifests/grafana-service
manifests/grafana-serviceAccount
manifests/grafana-serviceMonitor
manifests/kube-state-metrics-clusterRole
manifests/kube-state-metrics-clusterRoleBinding
manifests/kube-state-metrics-deployment
manifests/kube-state-metrics-role
manifests/kube-state-metrics-roleBinding
manifests/kube-state-metrics-service
manifests/kube-state-metrics-serviceAccount
manifests/kube-state-metrics-serviceMonitor
manifests/node-exporter-clusterRole
manifests/node-exporter-clusterRoleBinding
manifests/node-exporter-daemonset
manifests/node-exporter-service
manifests/node-exporter-serviceAccount
manifests/node-exporter-serviceMonitor
manifests/prometheus-adapter-apiService
manifests/prometheus-adapter-clusterRole
manifests/prometheus-adapter-clusterRoleAggregatedMetricsReader
manifests/prometheus-adapter-clusterRoleBinding
manifests/prometheus-adapter-clusterRoleBindingDelegator
manifests/prometheus-adapter-clusterRoleServerResources
manifests/prometheus-adapter-configMap
manifests/prometheus-adapter-deployment
manifests/prometheus-adapter-roleBindingAuthReader
manifests/prometheus-adapter-service
manifests/prometheus-adapter-serviceAccount
manifests/prometheus-clusterRole
manifests/prometheus-clusterRoleBinding
manifests/prometheus-operator-serviceMonitor
manifests/prometheus-prometheus
manifests/prometheus-roleBindingConfig
manifests/prometheus-roleBindingSpecificNamespaces
manifests/prometheus-roleConfig
manifests/prometheus-roleSpecificNamespaces
manifests/prometheus-rules
manifests/prometheus-service
manifests/prometheus-serviceAccount
manifests/prometheus-serviceMonitor
manifests/prometheus-serviceMonitorApiserver
manifests/prometheus-serviceMonitorCoreDNS
manifests/prometheus-serviceMonitorKubeControllerManager
manifests/prometheus-serviceMonitorKubeScheduler
manifests/prometheus-serviceMonitorKubelet
manifests/setup/0namespace-namespace
Opening output file: manifests/setup/0namespace-namespace: No such file or directory

interesting about namespaces. But the alertmanager-alertmanager file is there:

$ cat manifests/alertmanager-alertmanager 
{
   "apiVersion": "monitoring.coreos.com/v1",
   "kind": "Alertmanager",
   "metadata": {
      "labels": {
         "alertmanager": "main"
      },
      "name": "main",
      "namespace": "monitoring"
   },
   "spec": {
      "baseImage": "quay.io/prometheus/alertmanager",
      "nodeSelector": {
         "kubernetes.io/os": "linux"
      },
      "replicas": 3,
      "securityContext": {
         "fsGroup": 2000,
         "runAsNonRoot": true,
         "runAsUser": 1000
      },
      "serviceAccountName": "alertmanager-main",
      "version": "v0.18.0"
   }
}

Regarding coreutils, I'm running Debian Buster in WSL2 and here's what it says about it:

apt-cache search coreutils
coreutils - GNU core utilities

[bitva77]

Oh, namespace thing was me disabling that from the config.

But anyway, stepping through, command by command, if I cd into the manifests directly before running xargs, then this works:

jsonnet -J vendor -m manifests "${1-config.jsonnet}"; cd manifests; find . -type f | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml'

Not sure what it is about the normal pipe that it fails. Totally bizarre! must be something weird with WSL here.

Oh well, thanks for helping me through this weird one :)

[bitva77]

So! I think what's happening is the pipe and xargs execution is happening before WSL (the kernel?) has finished writing ./manifests/alertmanager-alertmanager and so the sh -c cat command in xargs is failing to find alertmanager-alertmanager because it truly doesn't exist yet. If I add a sleep statement, everything works fine :)

jsonnet -J vendor -m manifests "${1-config.jsonnet}" | xargs -I{} sh -c 'sleep .2; cat {} | gojsontoyaml > {}.yaml' -- {}