Charts: [bitnami/prometheus-operator] Thanos sidecar: WAL dir is not accessible

[halfmatthalfcat]

Which chart:

bitnami/prometheus-operator @ 0.29.3

Describe the bug

When booting up the thanos sidecar with a bucket configuration defined, thanos throws a

WAL dir is not accessible. Is this dir a TSDB directory? If yes it is shared with TSDB?: stat /prometheus/wal: no such file or directory

error and nothing is uploaded to the bucket. If the bucket configuration is removed from the chart, this error does not throw.

There was another ticket (#2251) were this issue was mentioned and one user (https://github.com/bitnami/charts/issues/2251#issuecomment-665079487) found a resolution however their fix (enabling prom persistence) didn't resolve my issue.

What I _think_ is happening is the pvc isn't being mounted fast enough for thanos to know there's data there and it never checks again, but I'm not versed enough in the internals to know exactly how that process operates.

Full log:

level=debug ts=2020-08-11T16:26:46.334411721Z caller=main.go:102 msg="maxprocs: Leaving GOMAXPROCS=[2]: CPU quota undefined"
level=info ts=2020-08-11T16:26:46.334570856Z caller=main.go:151 msg="Tracing will be disabled"
level=info ts=2020-08-11T16:26:46.334667503Z caller=options.go:23 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2020-08-11T16:26:46.334718401Z caller=grpc.go:92 service=gRPC/server component=sidecar msg="registering as gRPC StoreAPI and RulesAPI"
level=info ts=2020-08-11T16:26:46.334888389Z caller=factory.go:46 msg="loading bucket configuration"
level=error ts=2020-08-11T16:26:46.335253954Z caller=sidecar.go:256 err="WAL dir is not accessible. Is this dir a TSDB directory? If yes it is shared with TSDB?: stat /prometheus/wal: no such file or directory"
level=info ts=2020-08-11T16:26:46.33531248Z caller=sidecar.go:301 msg="starting sidecar"
level=info ts=2020-08-11T16:26:46.335448066Z caller=reloader.go:198 component=reloader msg="started watching config file and non-recursively rule dirs for changes" cfg= out= dirs=
level=info ts=2020-08-11T16:26:46.335660268Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2020-08-11T16:26:46.335905608Z caller=grpc.go:119 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901
level=info ts=2020-08-11T16:26:46.336064001Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2020-08-11T16:26:46.336080431Z caller=http.go:56 service=http/server component=sidecar msg="listening for requests and metrics" address=0.0.0.0:10902
level=warn ts=2020-08-11T16:26:46.340443739Z caller=sidecar.go:313 msg="failed to get Prometheus flags. Is Prometheus running? Retrying" err="request config against http://localhost:9090/api/v1/status/flags: Get \"http://localhost:9090/api/v1/status/flags\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=warn ts=2020-08-11T16:26:48.363957037Z caller=sidecar.go:313 msg="failed to get Prometheus flags. Is Prometheus running? Retrying" err="got non-200 response code: 503, response: Service Unavailable"
level=info ts=2020-08-11T16:26:50.342614221Z caller=sidecar.go:163 msg="successfully loaded prometheus external labels" external_labels="{prometheus=\"metrics/metrics-prometheus-operato-prometheus\", prometheus_replica=\"prometheus-metrics-prometheus-operato-prometheus-0\"}"
level=info ts=2020-08-11T16:26:50.342655702Z caller=intrumentation.go:48 msg="changing probe status" status=ready

To Reproduce
Steps to reproduce the behavior:

1. Install the 0.29.3 prometheus-operator helm chart with a bucket configuration
2. View the thanos sidecar logs

Expected behavior

The thanos sidecar should operate as normal when a bucket configuration is enabled and ship logs.

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:16:51Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5", GitCommit:"e0fccafd69541e3750d460ba0f9743b90336f24f", GitTreeState:"clean", BuildDate:"2020-04-16T11:35:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

[halfmatthalfcat]

I'm starting to narrow down the issue...prometheus is mounting the pvc with the subpath prometheus-db, which I see comes from here https://github.com/bitnami/charts/blob/1296de690e8a9712cbb65143723465ac60401e17/bitnami/prometheus-operator/templates/prometheus/prometheus.yaml#L223

However, it looks like subpathing this defaults to true (excluding the subpath).

My setup looks like:

prometheus:
  enabled: true
  disableCompaction: true
  persistence:
    enabled: true
  thanos:
    create: true
    image:
      repository: thanosio/thanos
      tag: v0.14.0
    extraArgs:
      - --log.level=debug
    objectStorageConfig:
      secretKey: thanos.yaml
      secretName: thanos-objstore-config

Ive been using the thanosio container to debug.

[halfmatthalfcat]

Actually, this issue with with the prometheus container itself, not thanos. Prometheus is mounting at subpath /prometheus-db

[marcosbc]

Thanks for reporting! It looks like indeed #3300 is linked to this issue. Adding on-hold tag until it is resolved.

[dbrennand]

Hey guys.
Thanks @halfmatthalfcat for doing the heavy lifting and debugging. I am also experiencing this same issue.

[dbrennand]

Hey @halfmatthalfcat I have deployed the latest chart version prometheus-operator-0.31.0 which contains fix #3300. I am still seeing the log error:
level=error ts=2020-08-11T16:26:46.335253954Z caller=sidecar.go:256 err="WAL dir is not accessible. Is this dir a TSDB directory? If yes it is shared with TSDB?: stat /prometheus/wal: no such file or directory"

However, the subpaths are correct now. Hopefully Thanos will still attempt to upload metrics after 2hrs. I will report back.

[dbrennand]

Bad news. No metrics are being uploaded. Can you verify @halfmatthalfcat ?

[dbrennand]

Success! 👍 @halfmatthalfcat

level=info ts=2020-08-18T11:59:20.441801723Z caller=shipper.go:361 msg="upload new block" id=01EG0PZ9KTQKSBA1PPFQQ2QJ91

Took sometime but looks like blocks are being uploaded by the sidecar. However, I am not sure if it was an action I performed which made a difference.

Around the 2HR pod lifetime mark there were no logs being uploaded and I had the WAL related error in the logs so I deleted the Prometheus pods and they were re-created (the WAL directory error didn't occur). Not sure whether this made the difference or not.