Loki: TLS support for loki and promtail

[Sh4d1]

Is your feature request related to a problem? Please describe.
I don't see TLS support anywhere, is it currently possible?

Describe the solution you'd like
If not, will/could it be implemented?
Describe alternatives you've considered
Or shoud we (the users) setup a reverse proxy managing TLS?

[daixiang0]

Now not possible, if you need this now, can use nginx to do it.

[Sh4d1]

@daixiang0 OK thanks! Ans is it planned in the future?

However it seems I cannot configure promtail with TLS certificats as well, so I need a external middleware that will set up the certificates?

[daixiang0]

For now you have to do like you said.

[cyriltovena]

@Sh4d1 we do support https for promtail. You just need to set correctly the configuration url as such:

client:
  url: https://mydomain.com/api/prom/push

If you are using custom root CA, you should consider adding the CA where you are running promtail, we don't support insecure and I don't think we will.

Server side I don't think we will add this feature as there is a lot of reverse proxy offering this for you.

What we could do though is offer a reverse proxy activation with TLS secret variable via helm to ease this for you.

Feel free to open an issue for that.

[r3kzi]

@Kuqd "we do support https for promtail" is only half right :-)

what about mTLS for Promtail <-> Loki? Can i configure Promtail to use a client certificate?

And i don't see a problem with insecure. Even Prometheus does it: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config

[cyriltovena]

Correct, still there is softwares solving that specific problem like consul and istio, If you're not in k8s that could be more difficult, there is a PR open especially for that let's see.

[r3kzi]

Yeah not really - unfortunately. I've already tried that. Loki/Promtail along with Istio in k8s.

When i deploy them along with Istio Envoy Sidecars...

Promtail gives me:

level=error ts=2019-05-02T09:16:23.403938144Z caller=client.go:186 msg="final error sending batch" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: <nil>"

Loki gives me:

evel=warn ts=2019-05-02T09:17:40.236262511Z caller=logging.go:49 traceID=765972c977acb165 msg="POST /api/prom/push (500) 224.321µs Response: \"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: <nil>\\n\" ws: false; Accept-Encoding: gzip; Content-Length: 1061; Content-Type: application/x-protobuf; User-Agent: Go-http-client/1.1; X-B3-Parentspanid: 0400748391a49ec0; X-B3-Sampled: 0; X-B3-Spanid: 6c7b2c533a4ce85f; X-B3-Traceid: 85c8078d4e6b88f50400748391a49ec0; X-Forwarded-Proto: http; X-Request-Id: d000a4ba-4996-4c14-a437-fa9891372647; "

I would be happy if you can tell me how to get this working! :-)

[AlessandroEmm]

I run into the same issue, and i'm wondering how promtail/loki are even affected by the surrounding mTLS.

[r3kzi]

@AlessandroEmm any progress so far? :-)

[tcolgate]

I'm looking at configuring for multiple clusters, mTLS is my usual preference for this. At the moment it looks like (promtail) -> (promtail -> loki) might be the best option?