Kube-prometheus: Can't add grafana dashboards

[Asgoret]

What did you do?
Add own grafana dashboards. If I add them via GUI all works perfectly, but when I tried adding them via ConfigMap it doesn't work. I downloaded this dashboard from grafana GUI, but if I try to use raw mode yaml file doesn't create at all

Code

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
    },
    prometheusAlerts+:: {
      groups+: (import 'dlgPromRules.json').groups,
    },
    grafanaDashboards+:: {
      'k8s-api.json': (import 'company/grafana/example.json'),
    },
  };
....

Error from Grafana
t=2020-05-14T17:20:46+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=0 file=/grafana-dashboard-definitions/0/k8s-api/k8s-api.json error="Dashboard title cannot be empty"

Additional information
I founded, that created json is not RFC standard. It has a lot of \\\.

[brancz]

Have you had a look at the docs here: https://github.com/coreos/kube-prometheus/blob/master/docs/developing-prometheus-rules-and-grafana-dashboards.md#pre-rendered-grafana-dashboards ?

[Asgoret]

@brancz Yeah, I read it about render (you can see it from code) and for some strange thing, the example from repo doesn't work either.

[brancz]

Your code uses grafanaDashboards not rawGrafanaDashboards (subtle difference, maybe we should call it out very explicitly in the docs?).

[atmosx]

Your code uses grafanaDashboards not rawGrafanaDashboards (subtle difference, maybe we should call it out very explicitly in the docs?).

I tried both, but still no dashboard generated:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-eks.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
      alertmanager+: {
        config: importstr 'config/dm-alertmanager.yaml',
      },
      prometheus+:: {
        namespaces: ["default", "kube-system", "ingress-nginx", "cert-manager", "kube-public"],
      },
      prometheusRules+:: {
        groups+: [
          {
            name: 'AWSCNI',
            rules: [
              {
                record: 'aws_eks_available_ip',
                expr: 'sum by(instance) (awscni_total_ip_addresses) - sum by(instance) (awscni_assigned_ip_addresses) < 10',
              },
            ],
          },
        ],
      },
      grafana+:: {
        config: {
          sections: {
            "auth.anonymous": {enabled: true, org_role: "Admin"},
          },
        },
      },
      rawGrafanaDashboards+:: {
        'ingress-nginx-controller.json': (importstr 'ingress.json'),
      },
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

All other definitions work as expected.

[brancz]

Which version of the grafana package are you using? (you should be able to find an exact version in your jsonnetfile.lock.json)

[atmosx]

Which version of the grafana package are you using? (you should be able to find an exact version in your jsonnetfile.lock.json)

Hello, thanks for the prompt reply, the version I'm using is the following:

  "version": 1,
  "dependencies": [
    {
      "source": {
        "git": {
          "remote": "https://github.com/brancz/kubernetes-grafana.git",
          "subdir": "grafana"
        }
      },
      "version": "57b4365eacda291b82e0d55ba7eec573a8198dda",
      "sum": "92DWADwGjnCfpZaL7Q07C0GZayxBziGla/O03qWea34="
    },
...
    {
      "source": {
        "git": {
          "remote": "https://github.com/grafana/grafonnet-lib.git",
          "subdir": "grafonnet"
        }
      },
      "version": "5736b62831d779e28a8344646aee1f72b1fa1d90",
      "sum": "ch97Uqauz7z+9mkOwzRz6JErxgWcQlfuJEEg+XHEadg="
    }
...
    {
      "source": {
        "git": {
          "remote": "https://github.com/grafana/jsonnet-libs.git",
          "subdir": "grafana-builder"
        }
      },
      "version": "811ccb022bc2bdcd0b8281ed0a0c858c63e20542",
      "sum": "slxrtftVDiTlQK22ertdfrg4Epnq97gdrLI63ftUfaE="
    },

I'm using kube-prometheus release-v4.0.

[brancz]

Looking at the kubernetes-grafana package code, I think our docs might be wrong, it seems like the rawDashboards configuration is actually nested in the Grafana configuration, so more like this:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-eks.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
      alertmanager+: {
        config: importstr 'config/dm-alertmanager.yaml',
      },
      prometheus+:: {
        namespaces: ["default", "kube-system", "ingress-nginx", "cert-manager", "kube-public"],
      },
      prometheusRules+:: {
        groups+: [
          {
            name: 'AWSCNI',
            rules: [
              {
                record: 'aws_eks_available_ip',
                expr: 'sum by(instance) (awscni_total_ip_addresses) - sum by(instance) (awscni_assigned_ip_addresses) < 10',
              },
            ],
          },
        ],
      },
      grafana+:: {
        rawDashboards+:: {
          'ingress-nginx-controller.json': (importstr 'ingress.json'),
        },
        config: {
          sections: {
            "auth.anonymous": {enabled: true, org_role: "Admin"},
          },
        },
      },
    },
  };

Could you give that a try, and if it works adapt the docs? :)

[Asgoret]

@brancz HI! Sorry for the long response.
My code works:

    grafanaDashboards+:: {
      'k8s-api.json': (import 'grafana/k8s-api.json'),
      'k8s-resources-dev.json': (import 'grafana/k8s-resources-dev.json'),
      'k8s-resources-prod.json': (import 'grafana/k8s-resources-prod.json'),
    },

UPD: If there any difference between grafanaDashboards & rawGrafanaDashboards it's will be great get it from the documentation.

[paulfantom]

Closing as this appears to be already fixed. Please reopen if this is still an issue.