Charts: Unable to send alerts from Grafana after setting SMTP settings in values.yaml file

[JoshLefebvre]

I've installed grafana using helm using a custom values.yaml file as follows:
helm install -f values.yaml stable/grafana --name grafana --namespace monitoring

I've added the following section to my values.yaml file to configure my smtp settings:

smtp:
  enabled: true
  host: "smtp.gmail.com"
  user: "[email protected]"
  password: "XXX"

I've validated that these settings are valid by using a test application to send emails. However when trying to set up notifications in grafana GUI i get the following log errors:

t=2019-02-19T18:48:18+0000 lvl=info msg="Sending alert notification to" logger=alerting.notifier.email addresses="[[email protected]]"
t=2019-02-19T18:48:18+0000 lvl=eror msg="Failed to send alert notification email" logger=alerting.notifier.email error="SMTP not configured, check your grafana.ini config file's [smtp] section."
t=2019-02-19T18:48:18+0000 lvl=eror msg="failed to send notification" logger=alerting.notifier id=0 error="SMTP not configured, check your grafana.ini config file's [smtp] section."

Does updating the smtp settings in the value file not automatically update the grafana.ini file when deploying? There has also been mention about using secrets to set SMTP settings but I have yet to find any documentation showing how to set a secret

[vsliouniaev]

Grafana is not able to reload configurations without a complete restart. You will need to delete the pod and recreate it.

To clarify, this isn't a chart limitation so much as a grafana binary limitation.
In my case, I invoke helm with prometheus-operator.grafana.podAnnotations.helmdeploytime: $(cat /proc/sys/kernel/random/uuid) which will force pod re-creation each time helm upgrade is called

[JoshLefebvre]

Just completely deleted my grafana install and purged it and then reinstalled it using helm and my values.yaml file containing my SMPT settings and still getting the same error message in the logs. I even did an exec -it into my contaienr and tried manually setting the env variables GF_SMTP_HOST GF_SMTP_ENABLED etc.... and still the same thing. While inside my container i checked my grafana.ini file and it did not contain any of my SMTP settings...

[vsliouniaev]

Can you post your entire values file?
Here's the place where grafana.ini configuration is added.
https://github.com/helm/charts/blob/master/stable/grafana/templates/configmap.yaml#L15

You are probably missing this info https://docs.helm.sh/chart_template_guide/#overriding-values-of-a-child-chart

Here is a sample configuration chunk for configuring the grafana sub-chart:

grafana:
  grafana.ini:
    users:
      viewers_can_edit: true
    auth:
      disable_login_form: true
      disable_signout_menu: true
    auth.anonymous:
      enabled: true
      org_name: Main Org.
      org_role: Viewer

[JoshLefebvre]

It's exactlly what exists in the current stable/grafana/values.yaml with the SMTP section modified to not use secrets and instead store password in plain text (for now).

rbac:
  create: true
  pspEnabled: true
  pspUseAppArmor: true
  namespaced: false
serviceAccount:
  create: true
  name:

replicas: 1

deploymentStrategy: RollingUpdate

readinessProbe:
  httpGet:
    path: /api/health
    port: 3000

livenessProbe:
  httpGet:
    path: /api/health
    port: 3000
  initialDelaySeconds: 60
  timeoutSeconds: 30
  failureThreshold: 10

image:
  repository: grafana/grafana
  tag: 5.4.3
  pullPolicy: IfNotPresent

  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistrKeySecretName

securityContext:
  runAsUser: 472
  fsGroup: 472


extraConfigmapMounts: []
  # - name: certs-configmap
  #   mountPath: /etc/grafana/ssl/
  #   configMap: certs-configmap
  #   readOnly: true


## Assign a PriorityClassName to pods if set
# priorityClassName:

downloadDashboardsImage:
  repository: appropriate/curl
  tag: latest
  pullPolicy: IfNotPresent

chownDataImage:
  repository: busybox
  tag: 1.30.0
  pullPolicy: IfNotPresent

## Pod Annotations
# podAnnotations: {}

## Deployment annotations
# annotations: {}

## Expose the grafana service to be accessed from outside the cluster (LoadBalancer service).
## or access it from within the cluster (ClusterIP service). Set the service type and the port to serve it.
## ref: http://kubernetes.io/docs/user-guide/services/
##
service:
  type: ClusterIP
  port: 80
  annotations: {}
  labels: {}

ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  labels: {}
  path: /
  hosts:
    - chart-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources: {}
#  limits:
#    cpu: 100m
#    memory: 128Mi
#  requests:
#    cpu: 100m
#    memory: 128Mi

## Node labels for pod assignment
## ref: https://kubernetes.io/docs/user-guide/node-selection/
#
nodeSelector: {}

## Tolerations for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: []

## Affinity for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}

## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  enabled: false
  initChownData: true
  # storageClassName: default
  accessModes:
    - ReadWriteOnce
  size: 10Gi
  # annotations: {}
  # subPath: ""
  # existingClaim:

# Administrator credentials when not using an existing secret (see below)
adminUser: admin
# adminPassword: strongpassword

# Use an existing secret for the admin user.
admin:
  existingSecret: ""
  userKey: admin-user
  passwordKey: admin-password

## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:

## Extra environment variables that will be pass onto deployment pods
env: {}

## The name of a secret in the same kubernetes namespace which contain values to be added to the environment
## This can be useful for auth tokens, etc
envFromSecret: ""

## Additional grafana server secret mounts
# Defines additional mounts with secrets. Secrets must be manually created in the namespace.
extraSecretMounts: []
  # - name: secret-files
  #   mountPath: /etc/secrets
  #   secretName: grafana-secret-files
  #   readOnly: true

## Additional grafana server volume mounts
# Defines additional volume mounts.
extraVolumeMounts: []
  # - name: extra-volume
  #   mountPath: /mnt/volume
  #   readOnly: true
  #   existingClaim: volume-claim

## Pass the plugins you want installed as a list.
##
plugins: []
  # - digrich-bubblechart-panel
  # - grafana-clock-panel

## Configure grafana datasources
## ref: http://docs.grafana.org/administration/provisioning/#datasources
##
datasources: {}
#  datasources.yaml:
#    apiVersion: 1
#    datasources:
#    - name: Prometheus
#      type: prometheus
#      url: http://prometheus-prometheus-server
#      access: proxy
#      isDefault: true

## Configure grafana dashboard providers
## ref: http://docs.grafana.org/administration/provisioning/#dashboards
##
## `path` must be /var/lib/grafana/dashboards/<provider_name>
##
dashboardProviders: {}
#  dashboardproviders.yaml:
#    apiVersion: 1
#    providers:
#    - name: 'default'
#      orgId: 1
#      folder: ''
#      type: file
#      disableDeletion: false
#      editable: true
#      options:
#        path: /var/lib/grafana/dashboards/default

## Configure grafana dashboard to import
## NOTE: To use dashboards you must also enable/configure dashboardProviders
## ref: https://grafana.com/dashboards
##
## dashboards per provider, use provider name as key.
##
dashboards: {}
#  default:
#    some-dashboard:
#      json: dashboards/custom-dashboard.json
#    prometheus-stats:
#      gnetId: 2
#      revision: 2
#      datasource: Prometheus
#    local-dashboard:
#      url: https://example.com/repository/test.json
#    local-dashboard-base64:
#      url: https://example.com/repository/test-b64.json
#      b64content: true

## Reference to external ConfigMap per provider. Use provider name as key and ConfiMap name as value.
## A provider dashboards must be defined either by external ConfigMaps or in values.yaml, not in both.
## ConfigMap data example:
##
## data:
##   example-dashboard.json: |
##     RAW_JSON
##
dashboardsConfigMaps: {}
#  default: ""

## Grafana's primary configuration
## NOTE: values in map will be converted to ini format
## ref: http://docs.grafana.org/installation/configuration/
##
grafana.ini:
  paths:
    data: /var/lib/grafana/data
    logs: /var/log/grafana
    plugins: /var/lib/grafana/plugins
    provisioning: /etc/grafana/provisioning
  analytics:
    check_for_updates: true
  log:
    mode: console
  grafana_net:
    url: https://grafana.net
## LDAP Authentication can be enabled with the following values on grafana.ini
## NOTE: Grafana will fail to start if the value for ldap.toml is invalid
  # auth.ldap:
  #   enabled: true
  #   allow_sign_up: true
  #   config_file: /etc/grafana/ldap.toml

## Grafana's LDAP configuration
## Templated by the template in _helpers.tpl
## NOTE: To enable the grafana.ini must be configured with auth.ldap.enabled
## ref: http://docs.grafana.org/installation/configuration/#auth-ldap
## ref: http://docs.grafana.org/installation/ldap/#configuration
ldap:
  # `existingSecret` is a reference to an existing secret containing the ldap configuration
  # for Grafana in a key `ldap-toml`.
  existingSecret: ""
  # `config` is the content of `ldap.toml` that will be stored in the created secret
  config: ""
  # config: |-
  #   verbose_logging = true

  #   [[servers]]
  #   host = "my-ldap-server"
  #   port = 636
  #   use_ssl = true
  #   start_tls = false
  #   ssl_skip_verify = false
  #   bind_dn = "uid=%s,ou=users,dc=myorg,dc=com"

## Grafana's SMTP configuration
## NOTE: To enable, grafana.ini must be configured with smtp.enabled
## ref: http://docs.grafana.org/installation/configuration/#smtp
smtp:
  # `existingSecret` is a reference to an existing secret containing the smtp configuration
  # for Grafana.
  enabled: true
  host: "smtp.sendgrid.net"
  user: "[email protected]"
  password: "XXXX"

## Sidecars that collect the configmaps with specified label and stores the included files them into the respective folders
## Requires at least Grafana 5 to work and can't be used together with parameters dashboardProviders, datasources and dashboards
sidecar:
  image: kiwigrid/k8s-sidecar:0.0.10
  imagePullPolicy: IfNotPresent
  resources:
#   limits:
#     cpu: 100m
#     memory: 100Mi
#   requests:
#     cpu: 50m
#     memory: 50Mi
  dashboards:
    enabled: false
    # label that the configmaps with dashboards are marked with
    label: grafana_dashboard
    # folder in the pod that should hold the collected dashboards
    folder: /tmp/dashboards
    # If specified, the sidecar will search for dashboard config-maps inside this namespace.
    # Otherwise the namespace in which the sidecar is running will be used.
    # It's also possible to specify ALL to search in all namespaces
    searchNamespace: null
  datasources:
    enabled: false
    # label that the configmaps with datasources are marked with
    label: grafana_datasource
    # If specified, the sidecar will search for datasource config-maps inside this namespace.
    # Otherwise the namespace in which the sidecar is running will be used.
    # It's also possible to specify ALL to search in all namespaces
    searchNamespace: null

[JoshLefebvre]

Just as a test I set my admin password in plain text in the above values.yaml file and when exec into my grafana container and doing printenv to list all my environment variables my password displayed in plain text under GF_SECURITY_ADMIN_PASSWORD.

No env variables are being set from the SMTP settings I am providing in the values file.... possible bug? Or is this only able to be configured via a secret???

[vsliouniaev]

If you search for where the smtp settings are used in the chart you will see the configuration you are setting is templated to allow overriding the specific values that are given in the values file. This is done via environment variables and allows you to set a secure configuration.

Searching through the grafana ini reference, there is a section relating to configuring smtp as well, so I suggest you use this approach in your case

[JoshLefebvre]

Yes you are correct and it is now working. I suppose I was confused because there is an existing section in the values.yaml file for smtp that allows you to store a secret. This is where I was storing my smtp settings, when in fact they should have been stored under the grafana.ini section in the values file as you mentioned.

Thanks for your help!

[srkfru]

Hello @vsliouniaev,

Need your suggestion here, I am doing POC on Prometheus and Grafana to monitor Docker, Kubernetes, Application, etc. and planning to build a centralized monitoring solution for all. I am running Prometheus in each K8 environment and created the data source in grafana. I succeed to place monitoring for Kubernetes and spring-boot applications as of now. But got stuck at alerting part.
1> Do I go with Prometheus or Grafana to build an alerting solution?
2> I am planning to move with grafana alerting solution as i do have multiple prometheus environement and can't create multiple grafana server for each.
3> while i am doing email integration with BPPM tool, we need all email objects so that we can map that with BPPM.

Kindly help over here.

[jonnalagadda35153]

when in fact they should have been

Can you please post the completed values.yml, i am facing the smae issue with SMTP configuration

[1996Shubh]

Hey i am facing same issue @jonnalagadda35153
You solved?

[jonnalagadda35153]

Yes, @1996shubh
Update SMTP values at grafana.ini part of the script no the one at the bottom

grafana.ini:
paths:
data: /var/lib/grafana/data
logs: /var/log/grafana
plugins: /var/lib/grafana/plugins
provisioning: /etc/grafana/provisioning
analytics:
check_for_updates: true
log:
mode: console
grafana_net:
url: https://grafana.net
smtp:
enabled: true
host: "smtp.gmail.com:587"
user: "[email protected]"
password: "xxxxxxxxxx"
skip_verify: true
from_address: "[email protected]"
from_name: "Grafana"

Hey i am facing same issue @jonnalagadda35153
You solved?