Charts: [stable/grafana] Allow storing grafana.ini as a secret

[mraszplewicz]

Is your feature request related to a problem? Please describe.
Grafana.ini is currently stored as a configmap value. I have to configure auth.generic_oauth with client_id and client_secret and also I would like to configure external database connection with password authentication. Storing it in configmap is not the best idea...

Describe the solution you'd like
Be able to use existing secret as a grafana.ini source.

Describe alternatives you've considered
Alternatively you can provide separate options to configure database connection and auth.generic_oauth.

Additional context
https://github.com/helm/charts/issues/22175 can be related

[jaystile]

I concur. My deployment of grafana requires the use of a couple of secrets and passwords for oauth and datasources. Since everything is routed through the grafana.ini, I am looking for a good way to obfuscate those values.

As I'm researching, I'm seeing that you can specify overrides with environment overrides
https://grafana.com/docs/grafana/latest/installation/configuration/#configure-with-environment-variables This may be a good attack vector by putting the override in a Secret and then adding those secrets as environment properties in the pod.

Yes, it appears that you can generate a Secret and replace the values in the grafana.ini and datasources. I added my pairs to the envRenderSecrets section. This placed environmental values in the pod. I was then able to modify by values.yaml to look like this:

envRenderSecret:
  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: <client secret value>
  DS_PROMETHEUS_SECUREJSONDATA_HTTPHEADERVALUE1: Bearer <BASE64 encoded token>

The grafana.ini didn't do what I expected. I put a placeholder value in for the client_secret, but it was not overridden by the environmental property. When I used variable syntax, it worked.

grafana.ini:
  auth.generic_oauth:
    ...
    client_secret: ${GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET}

This snippet of the datasources also allowed replacement variables.

datasources:
  datasources.yaml:
    datasources:
      ...
      secureJsonData:
        httpHeaderValue1: ${DS_PROMETHEUS_SECUREJSONDATA_HTTPHEADERVALUE1}

[s-spindler]

We are facing the same issue here. We need to configure a client_secret and would prefer if it wouldn't show up in the clear...

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[xUnholy]

This stale bot is absolutely useless...

As mentioned in my other issue #22175 it's important that I can use kubernetes secrets to store these secrets and not require to have them in my values for helm to render. existingSecret is already a function that is supported in the helm chart itself. EG.

admin:
  existingSecret: 'grafana-admin-creds'
   userKey: admin-user
   passwordKey: admin-password

Something a long these lines would suffice and be a much better solution.

[xUnholy]

@davidkarlsen @mrueg @GMartinez-Sisti Sorry for tagging you all out of an act of desperation, as it appears creating an issue may not be tracked/monitored in this monolithic repo with ease.

Is this something on the current roadmap?

[GMartinez-Sisti]

I'm not a maintainer but I think I can help.

Looks like there is already a way to perform this in a secure fashion without the need to change this chart or use environment variables for secrets.

1. Grafana config supports file providers for specific values:
   > file reads a file from the filesystem. It trims whitespace from the beginning and the end of files. The database password in the following example would be replaced by the content of the /etc/secrets/gf_sql_password file:

[database]
password = $__file{/etc/secrets/gf_sql_password}

1. Use the feature extraSecretMounts (Additional grafana server secret mounts) to mount your secrets, and go from there.

You just need to check where the extraSecretMounts will be available to specify them in grafana.ini.

[xUnholy]

@GMartinez-Sisti thanks for the swift response!

That is indeed one way, I'm trying to get my head around how that looks, would you be able to provide an example if it's not too much effort?

Ideally, the existingSecret functionality would be the simplest model considering it's already used in a lot of spots already, but if your method works then that is good enough!

[GMartinez-Sisti]

Using an example for oauth, you could do something like this:

grafana.ini:

grafana.ini:
  [auth.generic_oauth]
  enabled = true
  client_id = $__file{/etc/secrets/auth_generic_oauth/client_id}
  client_secret = $__file{/etc/secrets/auth_generic_oauth/client_secret}

Existing secret, or created along with helm:

---
apiVersion: v1
kind: Secret
metadata:
  name: auth-generic-oauth-secret
type: Opaque
stringData:
  client_id: <value>
  client_secret: <value>

Config for extraSecretMounts

- extraSecretMounts:
  - name: auth-generic-oauth-secret-mount
     secretName: auth-generic-oauth-secret
     defaultMode: 0440
     mountPath: /etc/secrets/auth_generic_oauth
     readOnly: true

This should work (famous last words™), note that I wasn't able to deploy to test, but according to the chart these are the required configurations.

Please let me know if it worked, and I might send a PR to add this to the README.

EDIT: Changed _ to - in the resources names. Thank you @mostlyAtNight.

[mostlyAtNight]

Hi @GMartinez-Sisti ,

Thanks for your post - this just worked for me using grafana 7.1+ - you'll need to change _ to - in your secret definition else kube complains (mine did anyway).

Kind regards,

Pete

[xUnholy]

@GMartinez-Sisti Unfortunately, mine doesn't seem to work - perhaps I'm missing something, but here is my config:

https://github.com/raspbernetes/k8s-gitops/blob/620b5e1c39373b3a50ab09db06481f65cc26a239/namespaces/observability/prometheus-operator/prometheus-operator.yaml#L158

I get to the redirect URL however it looks like something to do with my secret not being mounted it would appear because notice this in my URL $__file{/etc/secrets/auth_generic_oauth/client_id} where I'd expect to see the client_id value.

https://github.com/login/oauth/authorize?access_type=online&client_id=%24__file%7B%2Fetc%2Fsecrets%2Fauth_generic_oauth%2Fclient_id%7D&redirect_uri=https%3A%2F%2Fgrafana.raspbernetes.com%2Flogin%2Fgithub&response_type=code&scope=user%3Aemail+read%3Aorg&state=vQnPYTCWeoVVikFjc8I1lSO8-yO2uPn-RPAuGf2pdNY%3D

[GMartinez-Sisti]

@GMartinez-Sisti Unfortunately, mine doesn't seem to work - perhaps I'm missing something, but here is my config:

https://github.com/raspbernetes/k8s-gitops/blob/620b5e1c39373b3a50ab09db06481f65cc26a239/namespaces/observability/prometheus-operator/prometheus-operator.yaml#L158

I get to the redirect URL however it looks like something to do with my secret not being mounted it would appear because notice this in my URL $__file{/etc/secrets/auth_generic_oauth/client_id} where I'd expect to see the client_id value.

https://github.com/login/oauth/authorize?access_type=online&client_id=%24__file%7B%2Fetc%2Fsecrets%2Fauth_generic_oauth%2Fclient_id%7D&redirect_uri=https%3A%2F%2Fgrafana.raspbernetes.com%2Flogin%2Fgithub&response_type=code&scope=user%3Aemail+read%3Aorg&state=vQnPYTCWeoVVikFjc8I1lSO8-yO2uPn-RPAuGf2pdNY%3D

You need to update auth_generic_oauth to auth-generic-oauth-secret, because that is the name of your configured secret in the mount.

EDIT: Nevermind what I said. You are specifying the mount path correctly. Check that the secret exists and has the keys client_id and client_secret. Then jump into the pod and check that those files are actually there.

[xUnholy]

Correct me if I'm wrong however the auth_generic_oauth is just an arbitrary mountPath as long as the mount path is correct in my assignment it should be fine.

I think my issue is because I'm not running Grafana 7.1.+ as I just noticed the prometheus-operator for some reason is running 7.0.3 for some reason even though I'm using the latest version which should be 7.1.+.

EDIT: I have checked the secrets are mounted correctly, I did find in the docs it mentions this feature Only available in Grafana 7.1+. so I'm thinking that is my issue, testing now and will confirm shortly!

[GMartinez-Sisti]

The File provider is referenced in the documentation for v7.0, so maybe it's not that.

I see you are creating grafana.ini from helm, maybe helm is quoting those values, making it literals, and Grafana doesn't know it has to interpolate those. Try using $$__file instead. 🤷

[xUnholy]

Interestingly, it worked in the sense it took me to the correct oauth github project but then it redirected me to grafana and said "invalid username and password".

Here are the logs

t=2020-08-16T11:34:39+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=***.***.***.*** time_ms=0 size=29 referer=
t=2020-08-16T11:34:42+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/github status=302 remote_addr=***.***.***.*** time_ms=0 size=304 referer=https://grafana.raspbernetes.com/login
t=2020-08-16T11:35:09+0000 lvl=info msg="state check" logger=oauth queryState=f2f9849c470a30f0c72efeaaf2c2b7238ce62619a7dca90a88d964ad90a4d946 cookieState=f2f9849c470a30f0c72efeaaf2c2b7238ce62619a7dca90a88d964ad90a4d946
t=2020-08-16T11:35:10+0000 lvl=warn msg="Not allowing oauth_github login, user not found in internal user database and allow signup = false"
t=2020-08-16T11:35:10+0000 lvl=eror msg="Invalid Username or Password" logger=context userId=0 orgId=0 uname=
t=2020-08-16T11:35:10+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/github status=302 remote_addr=***.***.***.*** time_ms=1796 size=29 referer=https://github.com/

[xUnholy]

Okay fixed it. I don't exactly understand why but I needed allow_sign_up: true which was alluded in the logs I sent above Not allowing oauth_github login, user not found in internal user database and allow signup = false

[scottrigby]

📢 This chart is deprecated: https://github.com/helm/charts/pull/23662
If still applicable, please re-open issues at: https://github.com/grafana/helm-charts