Loki: Custom S3 Endpoint not authenticating via HTTPS

[ThoreKr]

When trying to set up loki with a custom s3 endpoint (ceph rgw) we always get authentication issues.
As far as header logging and tcpdump have shown loki initially tries to connect via http to the rgw. This call does contain an authorization header.

We redirect from HTTP to HTTPS (301), causing the http client to strip the authentication header and resend the request, causing the request to fail.

This issue seems to be located here: https://github.com/weaveworks/common/blob/54b7e30527f846e1515fb5a85d0ff5674f05a267/aws/config.go#L48

As apparently there is currently no way to force a direct connection via HTTPS.

[chancez]

Note it looks like https does work in the latest version of weaveworks/common: https://github.com/weaveworks/common/blob/master/aws/config.go#L53

[chancez]

But currently loki is using:

[[constraint]]
  name = "github.com/weaveworks/common"
  source = "https://github.com/sandlis/weaveworks-common"
  branch = "server-listen-addr"

[cyriltovena]

There’s an issue to move back to master, feel free to take it.

Le mer. 25 sept. 2019 à 20:23, Chance Zibolski [e-mail] a
écrit :

But currently loki is using:

[[constraint]]
name = "github.com/weaveworks/common"
source = "https://github.com/sandlis/weaveworks-common"
branch = "server-listen-addr"

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/grafana/loki/issues/1051?email_source=notifications&email_token=AAIBF3KMORQO5CYGLEVKDKLQLP6HJA5CNFSM4I2IY62KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7T2SRA#issuecomment-535275844,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAIBF3NB2B5IR573N5H7YXTQLP6HJANCNFSM4I2IY62A
.

[cyriltovena]

https://github.com/grafana/loki/issues/945

[ThoreKr]

I created a loki build based on weaveworks/common#167 now it does connect to the radosgw on port 443, but sends an http package (with an Authorization header).

[ThoreKr]

Hum, it seems to be partially related to stuff still being placed underneath vendor/github,com/weavework/common which is still outdated.

Not sure what needs to be updated here.

[MrSaints]

We've encountered a similar problem using Loki with DigitalOcean Spaces.

[ThoreKr]

Unfortunately it seems like there is little hope to get this working someday soon.

[ThoreKr]

@MrSaints since #1226 has been merged a custom built version works for us.

[MrSaints]

Thanks for the update, and for your personal time needed to push this through 👍

[mrmassis]

Hi, sorry by reopen this issue.
I couldnt find the way to configure ceph rgw https in loki.
Anyone can help me?

[ThoreKr]

This is my storage config:

storage_config:
  boltdb:
    directory: /var/lib/loki/index

  aws:
    s3: "https://access_key:[email protected]:443/loki"
    s3forcepathstyle: true

I think the port can be omitted by now, maybe even the protocol, i think it has been changed to use https by default, but it doesn't hurt to enforce it.

[mrmassis]

Tkz so much!