Charts: [stable/grafana] unable to configure dashboards in values file

[previ]

I cannot define dashboard to be imported in grafana at install time.

Description:
Configuring dashboard to be auto imported (by dashboard id) at install time, they are ignored.

Helm: 2.13.1
Kubernetes: 1.13.5

Which chart: stable/grafana 3.5.11

What happened: dashboards configured are actually ignored.

What you expected to happen: dashboard configured to be impoterd to be actually imported.

How to reproduce it (as minimally and precisely as possible):
use the following lines in the values.yanl:

dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
    - name: 'default'
      orgId: 1
      folder: ''
      type: file
      disableDeletion: false
      editable: true
      options:
        path: /var/lib/grafana/dashboards/default

dashboards:
  default:
    kubernetes-cluster:
      gnetId: 7249
      datasource: Prometheus

Anything else we need to know:

[irasnyd]

I hit this same issue today.

FYI, this works with the stable/grafana chart version 3.5.4, but does not work in the latest version 3.5.12.

[irasnyd]

Further testing shows these results:

| Version | Result |
| --- | --- |
| 3.5.4 | Works Perfectly |
| 3.5.5 | Works Perfectly |
| 3.5.6 | YAML parse error (see below) |
| 3.5.7 | Grafana crashes on startup (see below) |
| 3.5.8 | Grafana crashes on startup (logs are identical to 3.5.7) |

stable/grafana version 3.5.6 YAML parse error:

$ helm upgrade --install my-grafana stable/grafana --version 3.5.6 -f ~/kube-metrics/grafana-values.yaml --namespace default 
UPGRADE FAILED
Error: YAML parse error on grafana/templates/deployment.yaml: error converting YAML to JSON: yaml: line 39: did not find expected key
Error: UPGRADE FAILED: YAML parse error on grafana/templates/deployment.yaml: error converting YAML to JSON: yaml: line 39: did not find expected key

stable/grafana version 3.5.7 crash logs:

$ k logs -f my-grafana-5c956c7965-m7qws 
t=2019-07-20T00:00:23+0000 lvl=info msg="Starting Grafana" logger=server version=6.2.4 commit=ee01897 branch=HEAD compiled=2019-06-18T10:39:19+0000
t=2019-07-20T00:00:23+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2019-07-20T00:00:23+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_USER=admin"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_PASSWORD=*********"
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
t=2019-07-20T00:00:23+0000 lvl=info msg="App mode production" logger=settings
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing SqlStore" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2019-07-20T00:00:23+0000 lvl=info msg="Starting DB migration" logger=migrator
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing HTTPServer" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing InternalMetricsService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing RemoteCache" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing QuotaService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing LoginService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing PluginManager" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Starting plugin search" logger=plugins
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing RenderingService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing AlertingService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing HooksService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing SearchService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing ServerLockService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing TracingService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing UsageStatsService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing CleanUpService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing NotificationService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server
t=2019-07-20T00:00:23+0000 lvl=eror msg="Cannot read directory" logger=provisioning.dashboard type=file name=provisioned error="stat /var/lib/grafana/dashboards/provisioned: no such file or directory"
t=2019-07-20T00:00:23+0000 lvl=eror msg="Failed to read content of symlinked path" logger=provisioning.dashboard type=file name=provisioned path=/var/lib/grafana/dashboards/provisioned error="lstat /var/lib/grafana/dashboards/provisioned: no such file or directory"
t=2019-07-20T00:00:23+0000 lvl=info msg="falling back to original path due to EvalSymlink/Abs failure" logger=provisioning.dashboard type=file name=provisioned
t=2019-07-20T00:00:23+0000 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Failed to provision dashboards: Failed to provision config provisioned: stat /var/lib/grafana/dashboards/provisioned: no such file or directory"

Here is the complete differences between the chart version 3.5.5 and 3.5.7:

$ diff -Nru grafana-3.5.{5,7}
diff -Nru grafana-3.5.5/Chart.yaml grafana-3.5.7/Chart.yaml
--- grafana-3.5.5/Chart.yaml    1969-12-31 16:00:00.000000000 -0800
+++ grafana-3.5.7/Chart.yaml    1969-12-31 16:00:00.000000000 -0800
@@ -15,4 +15,4 @@
 name: grafana
 sources:
 - https://github.com/grafana/grafana
-version: 3.5.5
+version: 3.5.7
diff -Nru grafana-3.5.5/templates/deployment.yaml grafana-3.5.7/templates/deployment.yaml
--- grafana-3.5.5/templates/deployment.yaml 1969-12-31 16:00:00.000000000 -0800
+++ grafana-3.5.7/templates/deployment.yaml 1969-12-31 16:00:00.000000000 -0800
@@ -75,7 +75,8 @@
         - name: download-dashboards
           image: "{{ .Values.downloadDashboardsImage.repository }}:{{ .Values.downloadDashboardsImage.tag }}"
           imagePullPolicy: {{ .Values.downloadDashboardsImage.pullPolicy }}
-          command: ["sh", "/etc/grafana/download_dashboards.sh"]
+          command: ["/bin/sh"]
+          args: [ "-c", "mkdir -p /var/lib/grafana/dashboards/default", "/etc/grafana/download_dashboards.sh" ]
           volumeMounts:
             - name: config
               mountPath: "/etc/grafana/download_dashboards.sh"

[obeyler]

this should be solve by https://github.com/helm/charts/pull/15702

[obeyler]

The PR #15702 is merged could you check if it closes this issue ?
Thanks by advance @previ

[previ]

Tested stable/grafana chart version 3.6.0, same values as before; now I get Init:Error at container download-dashboards starting.
Error message in init container log: /bin/sh: /etc/grafana/download_dashboards.sh: Permission denied.

[irasnyd]

I confirmed that deploying the Helm chart version 3.7.0 does not work. The download-dashboards initContainer gives the following log message:

$ kubectl logs -f my-grafana-6cf4b4c8b9-qfm2p -c download-dashboards
/bin/sh: /etc/grafana/download_dashboards.sh: Permission denied

This is because the /etc/grafana/download_dashboards.sh file is not executable. AFAICT, it is not possible to instruct Kubernetes to set the executable mode bits on a file which is mounted from a ConfigMap.

I did this instead, which works as expected (the dashboards are provisioned correctly):

diff --git a/stable/grafana/templates/deployment.yaml b/stable/grafana/templates/deployment.yaml
index 0e01d32e0f86..d82578a38aff 100644
--- a/stable/grafana/templates/deployment.yaml
+++ b/stable/grafana/templates/deployment.yaml
@@ -76,7 +76,7 @@ spec:
           image: "{{ .Values.downloadDashboardsImage.repository }}:{{ .Values.downloadDashboardsImage.tag }}"
           imagePullPolicy: {{ .Values.downloadDashboardsImage.pullPolicy }}
           command: ["/bin/sh"]
-          args: [ "-c", "mkdir -p /var/lib/grafana/dashboards/default && /etc/grafana/download_dashboards.sh" ]
+          args: [ "-c", "mkdir -p /var/lib/grafana/dashboards/default && /bin/sh /etc/grafana/download_dashboards.sh" ]
           volumeMounts:
             - name: config
               mountPath: "/etc/grafana/download_dashboards.sh"

[previ]

solved in 3.7.3