Charts: [stable/grafana] invalid api key issue while accessing UI

[aliartiza75]

Describe the bug

I am trying to implement SSO using keycloak for Grafana. For that purpose, I have disabled the default login page so that the user is only authenticated by keycloak dashboard.

I am currently using prometheus-operator to deploy Grafana. Grafana configurations are given below:

    grafana:
      annotations:
        authproxy.domain.com/enabled: "true"
        authproxy.domain.com/upstream-url: "http://127.0.0.1:3000"
        authproxy.domain.com/source-service-name: domain-monitoring-prometheus-operator-grafana
        authproxy.domain.com/redirection-url: "https://grafana.domain.com"
        authproxy.domain.com/listen: "0.0.0.0:35000"
      grafana.ini:
        auth.anonymous:
          enabled: true
        auth:
          disable_login_form: "false"
        auth.generic_oauth:
          enabled: true
          client_id: "domain-platform"
          client_secret: "secret"
          auth_url: "keycloak-ingress-url"
          api_url: "keycloak-ingress-url"
          token_url: "keycloak-ingress-url"
          allow_sign_up: "false"
      # to enable grafana side car dashboards
      sidecar:
        dashboards:
          enabled: true
      adminPassword: 'SAMPLE_PASS' # Replace with actual Password
      rbac:
        create: true
        # it will create rbac on namespace level
        namespaced: true
      ingress:
        enabled: "true"
        hosts:
          - grafana.monitoring.domain.com
        tls:
          - secretName: tls-certificate
            hosts:
              - grafana.monitoring.domain.com
        annotations:
          kubernetes.io/ingress.class: "ingress-class"
          ingress.kubernetes.io/rewrite-target: "/"
          ingress.kubernetes.io/force-ssl-redirect: "true"

Grafana service has following configurations:

"ports": [
      {
        "name": "service",
        "protocol": "TCP",
        "port": 80,
        "targetPort": 35000
      }
    ],

In service, I am trying to forward the request to keycloak-gateway port(35000) and in keycloak-gateway I have specified to forward the request to grafana (localhost:3000).

The problem is that here when I try to access the dashboard I get this error:

"message":"Invalid API key"}

Although I have tried to pass the following configurations but still no luck:

 auth.generic_oauth:
    enabled: true
    client_id: "domain-platform"
    client_secret: "secret"
    auth_url: "keycloak-ingress-url"
    api_url: "keycloak-ingress-url"
    token_url: "keycloak-ingress-url"
    allow_sign_up: "false"

I there a way to handle the invalid api key issue.

Version of Helm and Kubernetes:

Kubernetes:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-21T15:34:43Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10+IKS", GitCommit:"03a1dfa2328c01886a41cdd08e9b91bfff3a4b4d", GitTreeState:"clean", BuildDate:"2019-08-21T11:54:44Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

Helm

Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

Which chart:
prometheus-operator: 5.11.0

What happened:
discussed above

What you expected to happen:
I want to authenticate the user using keycloak instead of grafana default dashboard.

How to reproduce it (as minimally and precisely as possible):
I have discussed above

Anything else we need to know:

No

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[aliartiza75]

not stale

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[savon-noir]

i face the same issue. anyone fixed it?

[wann-ap]

Did anyone fixed it?

i face the same issue. anyone fixed it?

[savon-noir]

keycloak-gatekeeper adds a HTTP Authorization Header. To control this, add in the keycloak-gatekeeper config enable-authorization-header: false

see here: https://github.com/keycloak/keycloak-documentation/blob/master/securing_apps/topics/oidc/keycloak-gatekeeper.adoc#upstream-headers

This applies for keycloak-gatekeeper but the same logic should apply to you oidc enabled reverse proxy

[rvansa]

Hi, any progress on this?

[Morriz]

We are actually running two grafana instances, both with exact same ini (second one for istio). One is working fine and we are actually using the JWT to map into org and roles. The other one is behind the same proxy setup, but has this error. Strange.

We have our setup on github/com/redkubes/otomi-core. It is a big setup but it might be beneficial to start using it.