Thanos: sidecar: Unable to connect to Azure storage on 0.4.0

[sbocz]

Thanos, Prometheus and Golang version used

Docker image: improbable/thanos:v0.4.0

What happened

Thanos sidecar container is having errors resolving Azure storage endpoint when updating image to 0.4.0 from 0.3.1.

What you expected to happen

No errors, as nothing else changed.

How to reproduce it (as minimally and precisely as possible):

Run prometheus with a sidecar container

- image: improbable/thanos:v0.4.0
  name: thanos-sidecar
  args:
  - sidecar
  - --prometheus.url=http://127.0.0.1:9090/
  - --tsdb.path=/prometheus
  - --cluster.disable
  - --grpc-address=[$(POD_IP)]:10901
  - --objstore.config=$(OBJSTORE_CONFIG)
  env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: status.podIP
  - name: OBJSTORE_CONFIG
    valueFrom:
      secretKeyRef:
        key: thanos.yaml
        name: thanos-objstore-config
  ports:
  - containerPort: 10902
    name: http
    protocol: TCP
  - containerPort: 10901
    name: grpc
    protocol: TCP
  - containerPort: 10900
    name: cluster
    protocol: TCP

apiVersion: v1
kind: Secret
metadata:
  name: thanos-objstore-config
data:
  thanos.yaml: 
# Data shown here is not literal value
type: AZURE
config:
  storage_account: "thanosaccount"
  storage_account_key: "key"
  container: "thanoscontainer"
  endpoint: "https://thanosaccount.blob.core.windows.net/"



md5-f28bfddea9496a89aac97479e86cc664



level=info ts=2019-05-17T06:21:53.355636937Z caller=flags.go:87 msg="gossip is disabled"
level=info ts=2019-05-17T06:21:53.355912237Z caller=main.go:257 component=sidecar msg="disabled TLS, key and cert must be set to enable"
level=info ts=2019-05-17T06:21:53.355950837Z caller=factory.go:39 msg="loading bucket configuration"
sidecar command failed: create AZURE client: Azure API return unexpected error: *pipeline.pcError
: -> github.com/Azure/azure-pipeline-go/pipeline.newDefaultHTTPClientFactory.func1.1, /go/pkg/mod/github.com/!azure/[email protected]/pipeline/core.go:234
HTTP request failed

Put https://thanosaccount.https://thanosaccount.blob.core.windows.net/thanoscontainer?restype=container&timeout=61: dial tcp: lookup thanosaccount.https on 10.0.0.10:53: no such host

Anything else we need to know

This looks like an issue in the way the configuration is parsed to get the storage endpoint to call.

@vglafirov

[sbocz]

I _think_ the issue might be here https://github.com/improbable-eng/thanos/blob/master/pkg/objstore/azure/helpers.go#L33. Sorry I've never used golang before

[sbocz]

I updated from

config:
  storage_account: "thanosaccount"
  storage_account_key: "KEY"
  container: "thanoscontainer"
  endpoint: "https://thanosaccount.blob.core.windows.net/"

to

config:
  storage_account: "thanosaccount"
  storage_account_key: "KEY"
  container: "thanoscontainer"
  endpoint: ""

And things are working now. I'm honestly not sure why I was using endpoint: "https://thanosaccount.blob.core.windows.net/" in my old setup. Perhaps I'll update the docs to show what each field is for.

[crewtool]

Hi, i had the same configuration and after deleting endpoint it worked. I think docs still miss information about each field.

[bwplotka]

:+1: