Prometheus-operator: Using bearerTokenSecret doesn't include the Bearer Token in the Authentication Header

[ellen-lau]

I'm trying to use the bearerTokenSecret definition in my ServiceMonitor to scrape my target's /metrics endpoint with a Bearer Token (used for authentication) included in the Authorization Header. It looks like my target is getting scraped, but the token I'm including in the secret I provide is not being included in the Authorization Header. The endpoint configuration for my ServiceMonitor looks something like this:

spec:
  endpoints:
    - bearerTokenSecret:
        key: token
        name: auth-token-token-6kzmt
      interval: 60s
      port: 9080-tcp
      tlsConfig:
        insecureSkipVerify: true

When I look at my application's access logs (The format of the access log is: 1) User-Agent, 2) Authorization Header), the requests from Prometheus have nothing in the Authorization Header (represented by the dash).

Prometheus/2.7.1 -

In comparison, when I try to curl with the bearer token using -H "Authorization: Bearer <TOKEN>, I can see the Bearer Token I include in the curl request:

curl/7.29.0 Bearer asdUadf....

For other example authentication methods like the ServiceMonitor's basicAuth definition, I can see those credentials in the Authorization Header as well:

Prometheus/2.7.1 Basic asdUadf....

Not sure why using the bearerTokenSecret definition in the ServiceMonitor doesn't include the specified secret's token inside the Authorization Header, and not sure if this is a configuration issue or a bug.

[brancz]

Thank you for opening this issue!

Could you provide us with the generated config, that you can find under /config in the Prometheus UI? That will tell us whether, the configuration was provisioned correctly and or whether the token was correctly provisioned. (secrets are automatically redacted from the UI :) so it's safe to share the config in that form)

[ellen-lau]

Under /config for the Prometheus UI, this following is what is generated. The ServiceMonitor in question is the appsody/test-token monitor.

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    prometheus: prometheus-operator/prometheus
    prometheus_replica: prometheus-prometheus-0
alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
  alertmanagers:
  - kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - prometheus-operator
    scheme: http
    path_prefix: /
    timeout: 10s
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: alertmanager-main
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: web
      replacement: $1
      action: keep
rule_files:
- /etc/prometheus/rules/prometheus-prometheus-rulefiles-0/*.yaml
scrape_configs:
- job_name: appsody/test-token/0
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - appsody
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance]
    separator: ;
    regex: test-token
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_monitor_openliberty_io_enabled]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: 9080-tcp
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: 9080-tcp
    action: replace
- job_name: prometheus-operator/myapp-monitor/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - liberty
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: websphere-liberty
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: 9080-tcp
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: 9080-tcp
    action: replace

I don't see anything specific under the monitor's config when I have bearerTokenSecret configured in the ServiceMonitor. In comparison, I know when basicAuth is configured in the ServiceMonitor, the config will show something like:

  basic_auth:
    username: admin
    password: <secret>

Thanks!

[ellen-lau]

Any updates with this? Thanks!

[brancz]

It does appear that there is a bug, as this should have been templated into the resulting configuration. One thing though, could you let us know what version of Prometheus Operator you are using?

[ellen-lau]

I am using Prometheus Operator version 0.32.0, which is the default that is downloaded off of OLM.

[spencergilbert]

I'm also seeing this problem, also on v0.32.0 - would a newer version resolve this issue?

[ellen-lau]

In addition to the above question, will this issue be patched for 0.32.0?

[brancz]

I can't find a bugfix in the changelog about this, so I assume this is still a problem in the latest version, it would be helpful if someone could validate that though.

[ellen-lau]

Are there any updates with this? Will this be fixed in a later version or patched for v0.32.0? Thanks.

[fredleger]

I can confirm the same behavior with a fresh prometheus operator deployed through rancher monitoring setup.
@brancz what kind of info do you need to dig any further ?

In my case i started suspecting this in a very simple setup:

1. deploy prometheus operator
2. deploy stable/minio with helm 3 (and with the service monitor enabled in prom ns)
3. Check the targets in prom UI

This will generate the secret and the config needed to fetch the metrics that you can fetch with a simple curl from prom pods.
But that does not work with the prom scrapes.

[cryptarchnoble]

I am experiencing this issue as well. I'm attempting to monitor metrics from AWX, and have created a secret containing my bearer token, which is the only way to authenticate against AWX programmatically. I confirmed that I can use this token/secret to authenticate against /api/v2/metrics on AWX when using curl... but creating a ServiceMonitor and specifying the following configuration:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: awx-servicemonitor
  namespace: cattle-prometheus
  lables:
    io.cattle.field/appId: cluster-monitoring
    release: cluster-monitoring
    source: rancher-monitoring
spec:
  namespaceSelector:
    matchNames:
      - awx
  selector:
    matchLabels:
      name: awx-web-svc
  endpoints:
  - port: http
    path: /api/v2/metrics
    scheme: http
    tlsConfig:
      insecureSkipVerify: true 
    bearerTokenSecret:
      name: awx-monitor-token
      key: token
    interval: 10s
----
apiVersion: v1
data:
  token: (my base64 encoded bearer token)
kind: Secret
metadata:
  name: awx-monitor-token
  namespace: cattle-prometheus
---

... this results in a "server returned HTTP status 401 Unauthorized" when inspecting the targets inside Prometheus. It is as if Prometheus is not using the token. Has anyone found a workaround?

[kanadaj]

@brancz Can confirm on v0.38.0; can't go any higher since we're on OKD 3.11; not sure if we can upgrade to 0.40.0 since the readme says it needs Kubernetes 1.16.

[brancz]

@paulfantom if I'm not mistaken I think you made use of bearerTokenSecret recently no? Was there something specific that you did to make that work?

[kanadaj]

Not sure about Paul, but my current workaround is to mount the bearer token secret as a file inside the prometheus containers and use bearerTokenFile instead. It works, but it did require recycling the pods so all transient data is gone :/

[paulfantom]

My current setup is available at https://github.com/thaum-xyz/ankhmorpork/blob/master/apps/homeassistant/08_servicemonitor.yaml#L12. This is running prometheus-operator v0.40.0 and I didn't change anything (working OOTB).

I had one issue when encoding token value as it also encoded \n and authentication failed, but this was not related to prometheus-operator.

[cryptarchnoble]

Thanks for the example @paulfantom . I double checked, and your config looks very similar to mine (with the exception of using a SealedSecret instead of a regular Secret, but I assume this is not the issue). My Secret and ServiceMonitor share the same namespace as yours do... only real difference I can determine is that your Service shares the same namespace as your ServiceMonitor and Secret, whereas my Service is in a different namespace (and thus, I use the namespaceSelector option in the endpoint definition for my ServiceMonitor).

I confirmed that I have no trailing \n in my Secret (have created it with echo -n 'unencoded secret' | base64 as well as recreating it via rancher which I assume doesn't add a \n, and end up with the same base64 hash, so I think the secret is good. Furthermore, I can curl my endpoint using my unencoded secret and the curl works, so I know the Secret is good.

I'll give the bearerTokenFile a shot and see where that gets me...

[cryptarchnoble]

I have confirmed that by mounting my secret as a volume on the prometheus pod and then updating my ServiceMonitor to use "bearerTokenFile" instead of "bearerTokenSecret" Prometheus is now able to scrape my endpoint successfully. I did not modify my secret, so definitely something wrong with the way Prometheus Operator is using secrets...

For completeness, here's my updated ServiceMonitor definition:

```apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: awx-servicemonitor
namespace: cattle-prometheus-p-xqzxx
lables:
io.cattle.field/appId: cluster-monitoring
release: cluster-monitoring
source: rancher-monitoring
spec:
namespaceSelector:
matchNames:
- awx
selector:
matchLabels:
name: awx-web-svc
endpoints:
- port: http
path: /api/v2/metrics
scheme: HTTP
tlsConfig:
insecureSkipVerify: true
bearerTokenFile: /etc/prometheus/secrets/awx/token
interval: 10s

[cryptarchnoble]

I've been digging further on this, and here's some additional color:

• We're using Rancher, which offers prometheus operator as part of it's built in monitoring solution.
• We're using an "older" build of rancher: 2.2.10. This includes "rancher monitoring" version 0.0.3.
• The docker image of prometheus seems to be rancher/prom-prometheus:v2.7.1.
  https://hub.docker.com/layers/rancher/prom-prometheus/v2.7.1/images/sha256-b32864cfe5f7f5d146820ddc6dcc5f78f40d8ab6d55e80decd5fc222b16344dc?context=explore
• The docker image of prometheus operator seems to be rancher/coreos-prometheus-operator:v0.29.0
  https://hub.docker.com/layers/rancher/coreos-prometheus-operator/v0.29.0/images/sha256-709595f835167800bf67bd39b2f3cba2d3c02e1efde688eb7ffe0359aad2f305?context=explore

Maybe Tokens weren't supported in this release? When I look at the prometheus.env.yaml file under /etc/prometheus/config_out/ I do not see any reference to my bearer token in the endpoint config... but if I use a bearer token file, the config does appear.

Here's the section of my prometheus.env.yaml config for the bearer token ServiceMonitor:

- job_name: cattle-prometheus-p-xqzxx/awx-servicemonitor-token/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - awx
  scrape_interval: 10s
  metrics_path: /api/v2/metrics/
  scheme: HTTP
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_name
    regex: awx-web-svc
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: http

Here's the config with a bearer token file:

- job_name: cattle-prometheus-p-xqzxx/awx-servicemonitor/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - awx
  scrape_interval: 10s
  metrics_path: /api/v2/metrics
  scheme: HTTP
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /etc/prometheus/secrets/awx/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_name
    regex: awx-web-svc
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: http

[kanadaj]

@cryptarchnoble Since it can be repro'd on OKD 3.11, it's unlikely to be Rancher's fault.

[paulfantom]

@cryptarchnoble you are using too old version. Support for bearerTokenSecret was added in https://github.com/coreos/prometheus-operator/pull/2716 which is part of 0.34.0 release.

@kanadaj OpenShift 3.11 is using prometheus-operator in version 0.23.2 (plus some patches), which is too old to support bearerTokenSecret. Source: https://github.com/openshift/prometheus-operator/blob/release-3.11/VERSION

[kanadaj]

@cryptarchnoble you are using too old version. Support for bearerTokenSecret was added in #2716 which is part of 0.34.0 release.

@kanadaj OpenShift 3.11 is using prometheus-operator in version 0.23.2 (plus some patches), which is too old to support bearerTokenSecret. Source: https://github.com/openshift/prometheus-operator/blob/release-3.11/VERSION

I've manually updated it to 0.38.0 using the upstream Kubernetes OLM, not using the default operator version. Other than this issue, it works perfectly fine.

[paulfantom]

@kanadaj sounds to me like in your case root cause is different. Could you check if RBAC permissions allow prometheus-operator to read secret referenced in bearerTokenSecret?

[kanadaj]

@paulfantom It should already have that:

  - verbs:
      - '*'
    apiGroups:
      - ''
    resources:
      - configmaps
      - secrets

It's present in both as a Role and ClusterRole and bound to prometheus-operator in the right namespace.

[jizi]

Hi @paulfantom seems we have the same issue as @kanadaj on version 0.39.0.
We tried different setups:

1. prometheus-operator in monitoring ns
   deployment of monitored app X, secret, servicemonitor in ns X
2. prometheus-operator and secret in monitoring ns
   deployment of monitored app X, servicemonitor in ns X
3. prometheus-operator and secret and servicemonitor in monitoring ns
   deployment of monitored app X in ns X

But in all cases the generated prometheus config was without bearerTokenSecret definition. Something like this:

...
- job_name: monitoring/X/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - reportportal
  relabel_configs:
...

There should be no issue accessing the secrets by prometheus-operator:

$ kubectl auth can-i get secret --as=system:serviceaccount:monitoring:prometheus-operator
yes

Do you have any suggestions regarding the investigation? Tried to check logs of prometheus-operator but nothing suspicious can be seen there.

[jizi]

Sorry, my bad. I had bug in ServiceMonitor definition. So I can confirm it works in 0.39.0

[brancz]

@jizi could you share for everyone what the culprit was? Maybe it's helpful for the next person who encounters it and maybe we can do a better job validating the CR :)

[jizi]

I somehow made a shortcut checking the documentation (https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint) that v1.SecretKeySelector is the whole structure used in e.g. deployment to set the env:

          valueFrom:
            secretKeyRef:
              name: "secretX"
              key: "keyX"

and had there these two extra levels valueFrom and secretKeyRef.
So definitely my bad. The documentation is correct in this.

[kanadaj]

@jizi Shouldn't it be:

    bearerTokenSecret:
      name: "secretX"
      key: "keyX"

?

According to the CRD:

                  bearerTokenSecret:
                    description: >-
                      Secret to mount to read bearer token for scraping targets.
                      The secret needs to be in the same namespace as the
                      service monitor and accessible by the Prometheus Operator.
                    type: object
                    required:
                      - key
                    properties:
                      key:
                        description: >-
                          The key of the secret to select from.  Must be a valid
                          secret key.
                        type: string
                      name:
                        description: >-
                          Name of the referent. More info:
                          https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                          TODO: Add other useful fields. apiVersion, kind, uid?
                        type: string
                      optional:
                        description: Specify whether the Secret or its key must be defined
                        type: boolean

Also, secretKeyRef isn't v1.SecretKeySelector but v1.EnvVarSource and valueFrom is v1.EnvVar

[jizi]

Exactly, what I mentioned in my previous comment was the buggy version.

Wrong

valueFrom:
  secretKeyRef:
    name: "secretX"
    key: "keyX"

Corrrect

bearerTokenSecret:
  name: "secretX"
  key: "keyX"

With the correct version I was able to make it work in 0.39.0.

[spencergilbert]

I think I did the same as @jizi, I tried again today on 0.38.1 and it is working :man_shrugging:

[simonpasquier]

It seems that it works for anybody using Prometheus operator >= 0.38. This issue should probably be closed (cc @paulfantom).

[paulfantom]

I agree with @simonpasquier. Closing.

If you find problems with bearerTokenSecret, please comment or open a new issue.