loki: ability to serve off non root path

[adamdecaf]

It's somewhat common to deploy services like loki off the web root (i.e. infra.corp.io/loki) when used with auth proxies. I'm currently unable to find a configuration for doing so.

Prometheus offers the following config:

--web.external-url=https://infra.corp.io/prometheus/

Looking at the webworks server config I don't see support already baked in for that either. Do you know if that's true? Would this be accepted into loki? I'll try and submit a patch.

[tomwilkie]

This is something we’d be keen on, yes! I guess it probably belongs in weaveworks/common/server.

[bboreham]

Similar to https://github.com/cortexproject/cortex/issues/778 ?

[adamdecaf]

Yep. That issue looks to be the same goal as this one.

[adamdecaf]

It looks like the package for this is running off @tomwilkie's fork, is there work needed to merge those together?

[[constraint]]
  name = "github.com/weaveworks/common"
  source = "github.com/tomwilkie/weaveworks-common"
  branch = "dev"

[tomwilkie]

@adamdecaf sorry about that - need https://github.com/weaveworks/common/pull/146 merging before we can go back to master. Shouldn't be long now!

[tomwilkie]

146 is merged now - @gouthamve or @adamdecaf, want to update the vendoring?

[adamdecaf]

@tomwilkie Done! https://github.com/grafana/loki/pull/211

[ArthurSens]

Does the version v0.2.0 provides any command line flag like --web.external-url from prometheus already? Or is it comming out with version v0.3.0?

[adamdecaf]

There's -server.path-prefix=/loki/

[cloudam]

helm upgrade --install loki loki/loki-stack --namespace=monitoring -server.path-prefix=/loki/
Error: unknown shorthand flag: 's' in -server.path-prefix=/loki/

[ArthurSens]

helm upgrade --install loki loki/loki-stack --namespace=monitoring -server.path-prefix=/loki/
Error: unknown shorthand flag: 's' in -server.path-prefix=/loki/

shouldn't it be something like?:

helm upgrade --install loki loki/loki-stack --namespace=monitoring --set server.path-prefix=/loki/

[cyriltovena]

It should be --set loki.extraArgs.server.path-prefix=/loki/ but it doesn't work with object in helm. see https://github.com/grafana/loki/issues/1668#issuecomment-584406225

Final podspec contains this args:

- "-server=map[path-prefix:/loki/]" which is weird.

[y0zg]

What is the working solution for path-prefix? @cyriltovena @ArthurSens

--set loki.extraArgs.server.path-prefix=/loki/ ?

[adamdecaf]

We use -server.path-prefix=/loki/ for our Loki setups on version 1.5.0.

See: https://github.com/moov-io/infra/blob/6e3fac6749bea9b70b1fe68e3a717923d0506484/lib/infra/19-loki.yml#L113

[y0zg]

Solution works for me

extraArgs:
  server.path-prefix: /loki/

livenessProbe:
  httpGet:
    path: /loki/ready
    port: http-metrics
  initialDelaySeconds: 45

readinessProbe:
  httpGet:
    path: /loki/ready
    port: http-metrics
  initialDelaySeconds: 45

[y0zg]

@adamdecaf Yep, exactly, I wonder how CLI option could help as config is dependent on probes. How do you access /metrics endpoint? I still see inconsistency between different loki endpoints

[y0zg]

Sorted out

 http:
  - match:
    - uri:
        prefix: /loki
    rewrite:
      uri: /loki
    route:
    - destination:
        host: loki.test.svc.cluster.local
        port:
          number: 3100
  - match:
    - uri:
        prefix: /metrics
    rewrite:
      uri: /loki/metrics
    route:
    - destination:
        host: loki.test.svc.cluster.local
        port:
          number: 3100

[cyriltovena]

values file is the way to go ! --set is unhappy :/