Kube-prometheus: Dashboard to big for ConfigMap wrapper

[michaelpietzsch]

What did you do?
Made custom modified jsonnet grafana deployment adding multiple dashboards
What did you expect to see?
the Dasboard in grafana
What did you see instead? Under which circumstances?
Error when applying config map
The ConfigMap "grafana-dashboard-node-exporter-full" is invalid: metadata.annotations: Too long: must have at most 262144 characters
Environment
DEV

• Prometheus Operator version:

insert image tag or Git SHA here

• Kubernetes version information:
  v1.11.2
  insert output of kubectl version here
  Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-18T11:37:06Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, tectonic-installer, etc.

• Manifests:

[metalmatze]

Do you combine all dashboards into one ConfigMap? We actually improve that in brancz/kubernetes-grafana by having one ConfigMap per dashboard.

[michaelpietzsch]

Hi Matthias,

I am using the latest version. The system is currently generating one ConfigMap per dashboard.
Chances are the config map solution is limited due to the etcd limits.

This might be a more fundamental issue when something between config map and persistent storage is needed

regards
Michael

Am 22.10.2018 um 14:37 schrieb Matthias Loibl <[email protected][e-mail]>:

Do you combine all dashboards into one ConfigMap? We actually improve that in brancz/kubernetes-grafanahttps://github.com/brancz/kubernetes-grafana/ by having one ConfigMap per dashboard.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/coreos/prometheus-operator/issues/2033#issuecomment-431822931, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKwi_-upi2_j1Uh0shytcJl1sbDXV3pQks5unbwbgaJpZM4XwWg6.

[metalmatze]

Not sure how we can work around the size limits of ConfigMaps to be honest... :confused:
Maybe you can try to improve the dashboard so it also reduces its size?

[michaelpietzsch]

Well there could be the possibility to create some sort of additional shell script which uses a POST request towards the Grafana API to push the dashboard into the system.

http://docs.grafana.org/http_api/dashboard/

But its far from ideal and cumbersome. :(

Regards
Michael

Am 23.10.2018 um 13:34 schrieb Matthias Loibl <[email protected][e-mail]>:

Not sure how we can work around the size limits of ConfigMaps to be honest... 😕
Maybe you can try to improve the dashboard so it also reduces its size?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/coreos/prometheus-operator/issues/2033#issuecomment-432210204, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKwi_1uvPPRidB3CZlca46d5-rfBMP7Dks5unv7BgaJpZM4XwWg6.

[metalmatze]

Yes, a sidecar like this was used in the past by kube-prometheus too:
https://github.com/coreos/prometheus-operator/tree/release-0.19/contrib/grafana-watcher

Otherwise you could extend the Grafana image with your dashboard.

Either way, I don't think there's a way of using the ConfigMaps approach if the dashboard is that big.

[brancz]

Do you know which dashboard is the one that's causing this? A dashboard larger than 1Mb sounds like it would be rather bloated in the first place, improving that sounds like a good idea either way. Otherwise you could maybe look into other volume mounts like a git volume.

[eyalzek]

I don't really understand, as per:
https://github.com/coreos/prometheus-operator/issues/535
https://github.com/eedugon/grafana-dashboards-configmap-generator

there was some kind of a workaround integrated into the repo, which was merged:
https://github.com/coreos/prometheus-operator/pull/567/files

but it was removed from the repo for some reason? Is there no workaround for this problem using the latest version?

[brancz]

@eyalzek they are different problems. The configmap generator was there to bin-pack dashboards into configmaps, which was later migrated from python+bash scripts to jsonnet. The problem here is that a single dashboard is larger than the configmap size limit, which no bin-packing can solve as the item is larger than the bin to begin with :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[afirth]

@michaelpietzsch @metalmatze: Since @aslafy-z 's PR https://github.com/kiwigrid/k8s-sidecar/pull/16 was merged, this works great in the sidecar and is already supported by the helm chart. I believe the sidecar only supports one file/CM though.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-dashboard
  namespace: prometheus-operator
  labels:
    grafana_dashboard: "1"
data:
  # fetch https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/grafana/dashboards/nginx.json with the grafana sidecar
  nginx-ingress.json.url: https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/grafana/dashboards/nginx.json

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[brancz]

Closing due to inactivity.