Charts: [stable/grafana] Cannot download dashboards for Grafana

Created on 18 Mar 2019  路  25Comments  路  Source: helm/charts

Is this a request for help?: Could be, maybe I'm just using the chart wrong

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Version of Helm and Kubernetes:
Helm:

{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}

Kubernetes:

{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
stable/grafana

What happened:
I tried to configure various dashboards to be downloaded for the Grafana deployment. But the download-dashboards init-container cannot download the charts to /var/lib/grafana/dashboards/default, as /var/lib/grafana/dashboards doesn't exist.
I tried circumventing this problem by mounting a secret into /var/lib/grafana/dashboards/default/_ but as I couldn't change the permissions up to the folder, it still got permission denied errors.

What you expected to happen:
That the dashboards would be downloaded and Grafana would start normally.

How to reproduce it (as minimally and precisely as possible):
With [stable/grafana] the following helm values.yml:

ingress:
  enabled: true
  hosts:
    - grafana.k8s.cwrau.io
plugins:
  - grafana-piechart-panel
dashboards:
  default:
    gitlab-runner:
      gnetId: 8729
      revision: 3
      datasource: Prometheus
    prometheus-stats:
      gnetId: 2
      revision: 2
      datasource: Prometheus

Anything else we need to know:

lifecyclstale
Source
picture cwrau

Most helpful comment

@cemsbr
I had that error before, you have to add the provider:

grafana:
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: default
          orgId: 1
          folder:
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/default
picture cwrau on 15 May 2019
👍4

All 25 comments

This seems like a bug with the grafana chart, since prometheus-operator is just using that directly without modification. Could you please update the issue with that information, the folks maintaining the chart would be able to answer your question quicker than I can

vsliouniaev picture vsliouniaev on 18 Mar 2019

Done! Thanks for the quick response!

cwrau picture cwrau on 18 Mar 2019

did you enable the appropriate dashboard provider?
https://github.com/helm/charts/blob/master/stable/grafana/values.yaml#L244

maorfr picture maorfr on 24 Mar 2019

Actually I overlooked that, but even with that config:

plugins:
  - grafana-piechart-panel
dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: default
        orgId: 1
        folder:
        type: file
        disableDeletion: true
        editable: false
        options:
          path: /var/lib/grafana/dashboards/default
dashboards:
  default:
    gitlab-runner:
      gnetId: 8729
      revision: 3
    prometheus-stats:
      gnetId: 2
      revision: 2
    TraefikRealtimeStatsby:
      gnetId: 2240
      revision: 3
    traefik2:
      gnetId: 5851
      revision: 3
    traefik1:
      gnetId: 4475
      revision: 4

The dashboards get downloaded, but are not accessible

cwrau picture cwrau on 25 Mar 2019

I got the same issue here. Was about to put a fix in, but this PR seems to fix it correctly for me:

https://github.com/helm/charts/pull/12512

bvandewalle picture bvandewalle on 25 Mar 2019

The dashboards get downloaded, but are not accessible

+1

qi-zhou picture qi-zhou on 26 Mar 2019

@cwrau @bvandewalle @qi-zhou
this is likely a new issue from one of the recent chart updates.
is anyone willing to try out some older versions of the chart and update which is the first version that has this issue?

thanks a lot!

maorfr picture maorfr on 26 Mar 2019

I just tried multiple versions, including the latest 2.3.4, and all are working, maybe it really is a bug, or misconfiguration, in the prometheus-operator chart?

cwrau picture cwrau on 26 Mar 2019

i'm a bit confused.
are you experiencing this issue when using the grafana chart or when using the prometheus-operator chart?

maorfr picture maorfr on 26 Mar 2019

I have this problem when using the grafana chart inside the prometheus-operator chart.

I changed the subject, because @vsliouniaev indicated, that the prometheus-operator chart uses the grafana chart without change.

But maybe the merged default config from the prometheus-operator and my custom config clash?

I just viewed the merged config:

grafana:
  additionalDataSources: []
  admin:
    existingSecret: ""
    passwordKey: admin-password
    userKey: admin-user
  adminPassword: prom-operator
  adminUser: admin
  affinity: {}
  chownDataImage:
    pullPolicy: IfNotPresent
    repository: busybox
    tag: 1.30.0
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - disableDeletion: true
        editable: false
        folder: null
        name: default
        options:
          path: /var/lib/grafana/dashboards/default
        orgId: 1
        type: file
  dashboards:
    default:
      TraefikRealtimeStatsby:
        gnetId: 2240
        revision: 3
      gitlab-runner:
        gnetId: 8729
        revision: 3
      prometheus-stats:
        gnetId: 2
        revision: 2
      traefik1:
        gnetId: 4475
        revision: 4
      traefik2:
        gnetId: 5851
        revision: 3
  dashboardsConfigMaps: {}
  datasources: {}
  defaultDashboardsEnabled: true
  deploymentStrategy: RollingUpdate
  downloadDashboardsImage:
    pullPolicy: IfNotPresent
    repository: appropriate/curl
    tag: latest
  enabled: true
  env: {}
  envFromSecret: ""
  extraConfigmapMounts: []
  extraContainers: ""
  extraSecretMounts: []
  extraVolumeMounts: []
  global:
    imagePullSecrets: []
    rbac:
      create: true
      pspEnabled: true
  grafana.ini:
    analytics:
      check_for_updates: true
    grafana_net:
      url: https://grafana.net
    log:
      mode: console
    paths:
      data: /var/lib/grafana/data
      logs: /var/log/grafana
      plugins: /var/lib/grafana/plugins
      provisioning: /etc/grafana/provisioning
  image:
    pullPolicy: IfNotPresent
    repository: grafana/grafana
    tag: 6.0.0
  ingress:
    annotations: {}
    enabled: true
    hosts:
    - grafana.k8s.4allportal.net
    labels: {}
    path: /
    tls: []
  ldap:
    config: ""
    existingSecret: ""
  livenessProbe:
    failureThreshold: 10
    httpGet:
      path: /api/health
      port: 3000
    initialDelaySeconds: 60
    timeoutSeconds: 30
  nodeSelector: {}
  persistence:
    accessModes:
    - ReadWriteOnce
    enabled: false
    initChownData: true
    size: 10Gi
  plugins:
  - grafana-piechart-panel
  rbac:
    create: true
    namespaced: false
    pspEnabled: true
    pspUseAppArmor: true
  readinessProbe:
    httpGet:
      path: /api/health
      port: 3000
  replicas: 1
  resources: {}
  securityContext:
    fsGroup: 472
    runAsUser: 472
  service:
    annotations: {}
    labels: {}
    port: 80
    targetPort: 3000
    type: ClusterIP
  serviceAccount:
    create: true
    name: null
  serviceMonitor:
    interval: ""
    selfMonitor: true
  sidecar:
    dashboards:
      enabled: true
      folder: /tmp/dashboards
      label: grafana_dashboard
      searchNamespace: null
    datasources:
      enabled: true
      label: grafana_datasource
      searchNamespace: null
    image: kiwigrid/k8s-sidecar:0.0.11
    imagePullPolicy: IfNotPresent
    resources: {}
  smtp:
    existingSecret: ""
    passwordKey: password
    userKey: user
  tolerations: []

Maybe I have to change something in there?

cwrau picture cwrau on 26 Mar 2019

can you try to compare this with the grafana config and examine the diffs?
feel free to share it here!

maorfr picture maorfr on 26 Mar 2019

This would be the default grafana config for prometheus-operator:

grafana:
  additionalDataSources: []
  admin:
    existingSecret: ""
    passwordKey: admin-password
    userKey: admin-user
  adminPassword: prom-operator
  adminUser: admin
  affinity: {}
  chownDataImage:
    pullPolicy: IfNotPresent
    repository: busybox
    tag: 1.30.0
  dashboardProviders: {}
  dashboards: {}
  dashboardsConfigMaps: {}
  datasources: {}
  defaultDashboardsEnabled: true
  deploymentStrategy: RollingUpdate
  downloadDashboardsImage:
    pullPolicy: IfNotPresent
    repository: appropriate/curl
    tag: latest
  enabled: true
  env: {}
  envFromSecret: ""
  extraConfigmapMounts: []
  extraContainers: ""
  extraSecretMounts: []
  extraVolumeMounts: []
  global:
    imagePullSecrets: []
    rbac:
      create: true
      pspEnabled: true
  grafana.ini:
    analytics:
      check_for_updates: true
    grafana_net:
      url: https://grafana.net
    log:
      mode: console
    paths:
      data: /var/lib/grafana/data
      logs: /var/log/grafana
      plugins: /var/lib/grafana/plugins
      provisioning: /etc/grafana/provisioning
  image:
    pullPolicy: IfNotPresent
    repository: grafana/grafana
    tag: 6.0.0
  ldap:
    config: ""
    existingSecret: ""
  livenessProbe:
    failureThreshold: 10
    httpGet:
      path: /api/health
      port: 3000
    initialDelaySeconds: 60
    timeoutSeconds: 30
  nodeSelector: {}
  persistence:
    accessModes:
    - ReadWriteOnce
    enabled: false
    initChownData: true
    size: 10Gi
  plugins: []
  rbac:
    create: true
    namespaced: false
    pspEnabled: true
    pspUseAppArmor: true
  readinessProbe:
    httpGet:
      path: /api/health
      port: 3000
  replicas: 1
  resources: {}
  securityContext:
    fsGroup: 472
    runAsUser: 472
  service:
    annotations: {}
    labels: {}
    port: 80
    targetPort: 3000
    type: ClusterIP
  serviceAccount:
    create: true
    name: null
  serviceMonitor:
    interval: ""
    selfMonitor: true
  sidecar:
    dashboards:
      enabled: true
      folder: /tmp/dashboards
      label: grafana_dashboard
      searchNamespace: null
    datasources:
      enabled: true
      label: grafana_datasource
      searchNamespace: null
    image: kiwigrid/k8s-sidecar:0.0.11
    imagePullPolicy: IfNotPresent
    resources: {}
  smtp:
    existingSecret: ""
    passwordKey: password
    userKey: user
  tolerations: []

This is the merged one:

grafana:
  additionalDataSources: []
  admin:
    existingSecret: ""
    passwordKey: admin-password
    userKey: admin-user
  adminPassword: prom-operator
  adminUser: admin
  affinity: {}
  chownDataImage:
    pullPolicy: IfNotPresent
    repository: busybox
    tag: 1.30.0
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - disableDeletion: true
        editable: false
        folder: null
        name: default
        options:
          path: /var/lib/grafana/dashboards/default
        orgId: 1
        type: file
  dashboards:
    default:
      TraefikRealtimeStatsby:
        gnetId: 2240
        revision: 3
      gitlab-runner:
        gnetId: 8729
        revision: 3
      prometheus-stats:
        gnetId: 2
        revision: 2
      traefik1:
        gnetId: 4475
        revision: 4
      traefik2:
        gnetId: 5851
        revision: 3
  dashboardsConfigMaps: {}
  datasources: {}
  defaultDashboardsEnabled: true
  deploymentStrategy: RollingUpdate
  downloadDashboardsImage:
    pullPolicy: IfNotPresent
    repository: appropriate/curl
    tag: latest
  enabled: true
  env: {}
  envFromSecret: ""
  extraConfigmapMounts: []
  extraContainers: ""
  extraSecretMounts: []
  extraVolumeMounts: []
  global:
    imagePullSecrets: []
    rbac:
      create: true
      pspEnabled: true
  grafana.ini:
    analytics:
      check_for_updates: true
    grafana_net:
      url: https://grafana.net
    log:
      mode: console
    paths:
      data: /var/lib/grafana/data
      logs: /var/log/grafana
      plugins: /var/lib/grafana/plugins
      provisioning: /etc/grafana/provisioning
  image:
    pullPolicy: IfNotPresent
    repository: grafana/grafana
    tag: 6.0.0
  ingress:
    annotations: {}
    enabled: true
    hosts:
    - grafana.k8s.4allportal.net
    labels: {}
    path: /
    tls: []
  ldap:
    config: ""
    existingSecret: ""
  livenessProbe:
    failureThreshold: 10
    httpGet:
      path: /api/health
      port: 3000
    initialDelaySeconds: 60
    timeoutSeconds: 30
  nodeSelector: {}
  persistence:
    accessModes:
    - ReadWriteOnce
    enabled: false
    initChownData: true
    size: 10Gi
  plugins:
  - grafana-piechart-panel
  rbac:
    create: true
    namespaced: false
    pspEnabled: true
    pspUseAppArmor: true
  readinessProbe:
    httpGet:
      path: /api/health
      port: 3000
  replicas: 1
  resources: {}
  securityContext:
    fsGroup: 472
    runAsUser: 472
  service:
    annotations: {}
    labels: {}
    port: 80
    targetPort: 3000
    type: ClusterIP
  serviceAccount:
    create: true
    name: null
  serviceMonitor:
    interval: ""
    selfMonitor: true
  sidecar:
    dashboards:
      enabled: true
      folder: /tmp/dashboards
      label: grafana_dashboard
      searchNamespace: null
    datasources:
      enabled: true
      label: grafana_datasource
      searchNamespace: null
    image: kiwigrid/k8s-sidecar:0.0.11
    imagePullPolicy: IfNotPresent
    resources: {}
  smtp:
    existingSecret: ""
    passwordKey: password
    userKey: user
  tolerations: []

And this is the diff:

diff --git a/tmp/default.yml b/tmp/merged.yml
index 5fa04a6..301f9e1 100644
--- a/tmp/default.yml
+++ b/tmp/merged.yml
@@ -11,8 +11,35 @@ grafana:
     pullPolicy: IfNotPresent
     repository: busybox
     tag: 1.30.0
-  dashboardProviders: {}
-  dashboards: {}
+  dashboardProviders:
+    dashboardproviders.yaml:
+      apiVersion: 1
+      providers:
+      - disableDeletion: true
+        editable: false
+        folder: null
+        name: default
+        options:
+          path: /var/lib/grafana/dashboards/default
+        orgId: 1
+        type: file
+  dashboards:
+    default:
+      TraefikRealtimeStatsby:
+        gnetId: 2240
+        revision: 3
+      gitlab-runner:
+        gnetId: 8729
+        revision: 3
+      prometheus-stats:
+        gnetId: 2
+        revision: 2
+      traefik1:
+        gnetId: 4475
+        revision: 4
+      traefik2:
+        gnetId: 5851
+        revision: 3
   dashboardsConfigMaps: {}
   datasources: {}
   defaultDashboardsEnabled: true
@@ -49,6 +76,14 @@ grafana:
     pullPolicy: IfNotPresent
     repository: grafana/grafana
     tag: 6.0.0
+  ingress:
+    annotations: {}
+    enabled: true
+    hosts:
+    - grafana.k8s.4allportal.net
+    labels: {}
+    path: /
+    tls: []
   ldap:
     config: ""
     existingSecret: ""
@@ -66,7 +101,8 @@ grafana:
     enabled: false
     initChownData: true
     size: 10Gi
-  plugins: []
+  plugins:
+  - grafana-piechart-panel
   rbac:
     create: true
     namespaced: false
cwrau picture cwrau on 26 Mar 2019

The issue for me was initially that the chart loader added an extra | \n before the dashboard json. This is now fixed upstream. Thanks!

bvandewalle picture bvandewalle on 27 Mar 2019

is this issue still actual?

maorfr picture maorfr on 5 Apr 2019

The config I provided last is not working, if it's not a mistake on my end, then this issue is still relevant

I saw #12512, maybe that fixes this? How do I update a required chart?

cwrau picture cwrau on 5 Apr 2019

@cwrau The latest chart version already references the grafana chart with that fix in it

vsliouniaev picture vsliouniaev on 5 Apr 2019

The json files are in /var/lib/grafana/dashboards/default/, but are not getting imported into Grafana

Still same parameters as before

grafana:
  ingress:
    enabled: true
    hosts:
      - grafana.k8s.4allportal.net
  plugins:
    - grafana-piechart-panel
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: default
          orgId: 1
          folder:
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/default
  dashboards:
    default:
      gitlab-runner:
        gnetId: 8729
        revision: 3
      prometheus-stats:
        gnetId: 2
        revision: 2
      TraefikRealtimeStatsby:
        gnetId: 2240
        revision: 3
      traefik2:
        gnetId: 5851
        revision: 3
      traefik1:
        gnetId: 4475
        revision: 4
cwrau picture cwrau on 5 Apr 2019

I'm also using prometheus-operator with grafana; and am experiencing very similar problems; except that it doesn't seem to even download the json; when I ssh into the grafana pod I see that /var/lib/grafana doesn't even exist.

I'm using prometheus-operator-5.0.11 which relies on grafana-2.3.5;

Here're my grafana values overrides; which seem similar to @cwrau except that I specify to download json from a gist; does this seem to be part of the same issue?

  grafana:
    adminPassword: "admin"
    ingress:
      enabled: false
    persistence:
      storageClassName: "aws-ebs-retained"
      enabled: true
      accessModes: ["ReadWriteOnce"]
      size: 10Gi

    dashboardProviders:
      dashboardproviders.yaml:
        apiVersion: 1
        providers:
          - name: 'default'
            orgId: 1
            folder: ''
            type: file
            disableDeletion: true
            editable: false
            options:
              path: /var/lib/grafana/dashboards/default

    dashboards:
      default:
        galley:
          url: https://gist.githubusercontent.com/ChrisPenner/dc8f2b50b62eaf746ce306df4e610126/raw/446d15fe9b3f76ef4a0cdb6175a628c33aac0039/galley.json
ChrisPenner picture ChrisPenner on 11 Apr 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

stale[bot] picture stale[bot] on 11 May 2019

This issue still exists for me

cwrau picture cwrau on 13 May 2019
👍1

The pod cannot download dashboards:

/etc/grafana/download_dashboards.sh: line 8: can't create /var/lib/grafana/dashboards/default/haproxy.json: nonexistent directory
cemsbr picture cemsbr on 14 May 2019

@cemsbr
I had that error before, you have to add the provider:

grafana:
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: default
          orgId: 1
          folder:
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/default
cwrau picture cwrau on 15 May 2019
👍4

Hi,

I am trying to setup grafana with SMTP options enabled. below are the options configured.

grafana.ini:
smtp:
enabled: true
existingSecret: ""
host: smtpserver:port
paths:
data: /var/lib/grafana/data
logs: /var/log/grafana
plugins: /var/lib/grafana/plugins
provisioning: /etc/grafana/provisioning
analytics:
check_for_updates: true
log:
mode: console
grafana_net:
url: https://grafana.net

Post deployment, My grafana dashboard is not enabled with SMTP settings.
Can anyone let me know what is the incorrect configuration.

lavanyaanbuatgmaildotcom picture lavanyaanbuatgmaildotcom on 31 May 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

stale[bot] picture stale[bot] on 30 Jun 2019

This issue is being automatically closed due to inactivity.

stale[bot] picture stale[bot] on 14 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tdcox picture tdcox  路  3Comments
gajus picture gajus  路  3Comments
michas2 picture michas2  路  3Comments
onyxet picture onyxet  路  3Comments
dene14 picture dene14  路  3Comments