Thanos: Thanos store can't access S3 bucket after upgrade from v0.10.1 to v0.11.0-rc.0

[vears91]

Thanos, Prometheus and Golang version used:
Thanos store version 0.11.0-rc.0, go1.13.1

Object Storage Provider:
Amazon S3

What happened:
Upgraded Thanos store from v0.10.1 to v0.11.0-rc.0. Thanos store cannot access the bucket during start-up, showing "Access Denied" errors and exits with error code 1

What you expected to happen:
Thanos store starts successfully

How to reproduce it (as minimally and precisely as possible):
Run Thanos store v0.11.0-rc.0 using S3 as object storage and kube2iam for access

Full logs to relevant components:

level=debug ts=2020-02-25T09:32:57.959151515Z caller=main.go:104 msg="maxprocs: Updating GOMAXPROCS=[1]: using minimum allowed GOMAXPROCS"
level=info ts=2020-02-25T09:32:57.959392074Z caller=main.go:152 msg="Tracing will be disabled"
level=info ts=2020-02-25T09:32:57.959537528Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2020-02-25T09:32:57.959964535Z caller=inmemory.go:172 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=1073741824 maxItems=math.MaxInt64
level=info ts=2020-02-25T09:32:57.960224229Z caller=options.go:23 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2020-02-25T09:32:57.960538942Z caller=store.go:314 msg="starting store node"
level=info ts=2020-02-25T09:32:57.960783296Z caller=intrumentation.go:64 msg="changing probe status" status=healthy
level=info ts=2020-02-25T09:32:57.960814203Z caller=http.go:56 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2020-02-25T09:32:57.961163794Z caller=store.go:269 msg="initializing bucket store"
level=warn ts=2020-02-25T09:32:58.060664033Z caller=intrumentation.go:58 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: MetaFetcher: iter bucket: Access Denied"
level=info ts=2020-02-25T09:32:58.060778795Z caller=http.go:81 service=http/server component=store msg="internal server shutdown" err="bucket store initial sync: sync block: MetaFetcher: iter bucket: Access Denied"
level=info ts=2020-02-25T09:32:58.060809748Z caller=intrumentation.go:70 msg="changing probe status" status=not-healthy reason="bucket store initial sync: sync block: MetaFetcher: iter bucket: Access Denied"
level=warn ts=2020-02-25T09:32:58.060840622Z caller=intrumentation.go:58 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: MetaFetcher: iter bucket: Access Denied"
level=info ts=2020-02-25T09:32:58.060866542Z caller=grpc.go:124 service=gRPC/server component=store msg="gracefully stopping internal server"
level=info ts=2020-02-25T09:32:58.060897868Z caller=grpc.go:136 service=gRPC/server component=store msg="internal server shutdown" err="bucket store initial sync: sync block: MetaFetcher: iter bucket: Access Denied"
level=info ts=2020-02-25T09:32:58.060948033Z caller=intrumentation.go:52 msg="changing probe status" status=ready
level=info ts=2020-02-25T09:32:58.061036906Z caller=grpc.go:105 service=gRPC/server component=store msg="listening for StoreAPI gRPC" address=0.0.0.0:10901
level=error ts=2020-02-25T09:32:58.061092902Z caller=main.go:210 msg="running command failed" err="bucket store initial sync: sync block: MetaFetcher: iter bucket: Access Denied"

Anything else we need to know:
Running on Kubernetes using kube2iam

[metalmatze]

There was an update to the minio libarary between v0.10 and v0.11. Let's check that: https://github.com/thanos-io/thanos/pull/2033

[bigkraig]

I think this was fixed with https://github.com/minio/minio-go/pull/1224 and released in v6.0.47

[yeya24]

See the comment here https://github.com/thanos-io/thanos/issues/448#issuecomment-589807381. @metalmatze Seems we need to update minio-go as @bigkraig said.

[metalmatze]

Yeah. That was my intention too looking through their changelog.
If anyone wants to do a PR against the release branch I'm happy to review it. Otherwise I'll get to it tomorrow (just stopped working).

[bigkraig]

ill get a PR in

[bigkraig]

@metalmatze #2189

[metalmatze]

New RC is out.
It would be amazing if you could test this one! :relaxed:
https://github.com/thanos-io/thanos/releases/tag/v0.11.0-rc.1

[skrytt]

Hey, I was running into the same problem as the issue author after upgrading to Thanos Store v0.11.0-rc.0.
I have upgraded the affected deployment to use v0.11.0-rc.1 instead, now Thanos Store talks successfully to S3 with no Access Denied error :+1:
Thanks!

[bigkraig]

:+1:, been running it for about an hour now

[metalmatze]

You people are insane! So quickly testing it.
Thank you soooo much! :blush:

[bwplotka]

This was tricky to investigate! Thanks everyone involved! :heart: :hugs: