Kube-prometheus: Support for sharding dashboards to avoid configmap size limits

[e9169]

What is missing?

Support for dashboard sharding due to configmap size limits.

Why do we need it?

While deploying certain dashboards generated with the import option, I get "metadata.annotations: Too long: must have at most 262144 characters"

Environment

Kubernetes 1.15.5 and 1.14.8

kube-prometheus version:

Latest

Anything else we need to know?:

This is the PR that applies these changes to the libs: https://github.com/grafana/jsonnet-libs/pull/159/files

[brancz]

We already put each dashboard in its own config map, I think what you may be running into is a single dashboard being too big which we can’t really solve with what you linked I believe.

[e9169]

We already put each dashboard in its own config map, I think what you may be running into is a single dashboard being too big which we can’t really solve with what you linked I believe.

Hi @brancz No, that's really what they do with that PR and our issue involves a too big dashboard, yes. That PR fixes this issue, which describes perfectly what happens with a big dashboard:

https://github.com/grafana/tanka/issues/50

[brancz]

What you linked still shards full dashboards only, but does not address the fact that when _single_ dashboard is too large this still happens. We're already doing what you linked in a more primitive way, by not using up all the space available but just putting 1 dashboard into 1 configmap.

[e9169]

What you linked still shards full dashboards only, but does not address the fact that when _single_ dashboard is too large this still happens. We're already doing what you linked in a more primitive way, by not using up all the space available but just putting 1 dashboard into 1 configmap.

I don't fully understand your point, sorry. The change introduced in the PR and Issue mentioned above says " // Optionally shard dashboards into multiple config maps.", and this change is made only to address the error of annotation being too large for a SINGLE dashboard. In fact, we have many dashboards and only a few of them throw the error.

[brancz]

What you linked puts multiple dashboards (or all if sharding is disabled) into one configmap and when sharding is enabled it distributes the full dashboards across multiple config maps but doesn’t split dashboards themselves. At least I can’t see in the code where it does that. I can only spot where the dashboards choose their shard, and that process we already kind of do as we essentially have number of shards equal to the amount of dashboards.

[e9169]

What you linked puts multiple dashboards (or all if sharding is disabled) into one configmap and when sharding is enabled it distributes the full dashboards across multiple config maps but doesn’t split dashboards themselves. At least I can’t see in the code where it does that. I can only spot where the dashboards choose their shard, and that process we already kind of do as we essentially have number of shards equal to the amount of dashboards.

Thanks, now it's perfectly clear. I suppose there is no way then to split a big dashboard into different configmaps, right?

[brancz]

I am not aware of a way to do that. Maybe we can experiment with how much in size a dashboard would shrink when gzipped, and if it's significant maybe we can convince Grafana to allow loading gzipped dashboards? That's what we did when the Prometheus configuration became too huge and it hasn't been a problem since.

[GTB3NW]

I get this with https://grafana.com/grafana/dashboards/1860

Would be great to have another way to load big dashboards