Thanos: query-frontend: redirect loop when visiting bare path prefix like `/thanos`

Created on 16 Sep 2020  路  4Comments  路  Source: thanos-io/thanos

Thanos, Prometheus and Golang version used:

$ thanos --version
thanos, version 0.15.0 (branch: HEAD, revision: fbd14b49f95e7543883fedf9381a21127d4dda2b)
  build user:       root@d35b081d3e19
  build date:       20200907-09:29:21
  go version:       go1.14.2

What happened:

When using query-frontend with query that uses a path prefix, we encounter a 301 loop and ERR_TOO_MANY_REDIRECTS error when visiting the path prefix, such as http://localhost/thanos/ but a successful page load on http://localhost/thanos/graph. Visiting http://localhost/thanos/ on query correctly redirects to /thanos/graph

With curl:

$ curl "localhost:9093/thanos/"
<a href="/thanos/">Moved Permanently</a>.

$ curl "localhost:9092/thanos/"
<a href="/thanos/graph">Found</a>.

What you expected to happen: visiting bare path on query-frontend to 302 to path prefix + /graph

How to reproduce it (as minimally and precisely as possible):

  1. Run thanos query (port 9092) and query-frontend (port 9093) together with cli args:
thanos query --http-address=127.0.0.1:9092 --web.route-prefix=thanos --web.external-prefix=thanos ...

thanos query-frontend --http-address=127.0.0.1:9093 --query-frontend.compress-responses --query-frontend.downstream-url=http://127.0.0.1:9092 --query-range.split-interval=24h --query-range.max-retries-per-request=5 --query-frontend.log_queries_longer_than=5s --query-range.response-cache-config-file=/etc/thanos/range_cache.yml
  1. Compare request to query with path prefix:
$ curl -vsL http://localhost:9092/thanos | head
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9092 (#0)
> GET /thanos HTTP/1.1
> Host: localhost:9092
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Type: text/html; charset=utf-8
< Location: /thanos/
< Date: Wed, 16 Sep 2020 16:39:14 GMT
< Content-Length: 43
<
* Ignoring the response-body
{ [43 bytes data]
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:9092/thanos/'
* Found bundle for host localhost: 0x7f989e41c960 [can pipeline]
* Could pipeline, but not asked to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 9092 (#0)
> GET /thanos/ HTTP/1.1
> Host: localhost:9092
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 302 Found
< Content-Type: text/html; charset=utf-8
< Location: /thanos/graph
< Date: Wed, 16 Sep 2020 16:39:14 GMT
< Content-Length: 36
<
* Ignoring the response-body
{ [36 bytes data]
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:9092/thanos/graph'
* Found bundle for host localhost: 0x7f989e41c960 [can pipeline]
* Could pipeline, but not asked to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 9092 (#0)
> GET /thanos/graph HTTP/1.1
> Host: localhost:9092
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 16 Sep 2020 16:39:14 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
<
{ [5494 bytes data]
* Connection #0 to host localhost left intact
* Closing connection 0
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <meta name="robots" content="noindex,nofollow">
        <title>Thanos long term storage Prometheus solution</title>
        <link rel="shortcut icon" href="/thanos/static/img/favicon.ico?v=fbd14b49f95e7543883fedf9381a21127d4dda2b">
        <script src="/thanos/static/vendor/js/jquery-3.5.0.min.js?v=fbd14b49f95e7543883fedf9381a21127d4dda2b"></script>
        <script src="/thanos/static/vendor/js/popper.min.js?v=fbd14b49f95e7543883fedf9381a21127d4dda2b"></script>
        <script src="/thanos/static/vendor/bootstrap-4.1.3/js/bootstrap.min.js?v=fbd14b49f95e7543883fedf9381a21127d4dda2b"></script>
  1. Compare query above with query-frontend request with path prefix:
禄 curl -vsL http://localhost:9093/thanos | head
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9093 (#0)
> GET /thanos HTTP/1.1
> Host: localhost:9093
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Length: 43
< Content-Type: text/html; charset=utf-8
< Date: Wed, 16 Sep 2020 16:40:49 GMT
< Location: /thanos/
< Vary: Accept-Encoding
< Vary: Accept-Encoding
<
* Ignoring the response-body
{ [43 bytes data]
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:9093/thanos/'
* Found bundle for host localhost: 0x7f97f0c0d420 [can pipeline]
* Could pipeline, but not asked to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 9093 (#0)
> GET /thanos/ HTTP/1.1
> Host: localhost:9093
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Length: 43
< Content-Type: text/html; charset=utf-8
< Date: Wed, 16 Sep 2020 16:40:49 GMT
< Location: /thanos/
< Vary: Accept-Encoding
< Vary: Accept-Encoding
<
* Ignoring the response-body
{ [43 bytes data]
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:9093/thanos/'
* Found bundle for host localhost: 0x7f97f0c0d420 [can pipeline]
* Could pipeline, but not asked to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 9093 (#0)
> GET /thanos/ HTTP/1.1
> Host: localhost:9093
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Length: 43
< Content-Type: text/html; charset=utf-8
< Date: Wed, 16 Sep 2020 16:40:49 GMT
< Location: /thanos/
< Vary: Accept-Encoding
< Vary: Accept-Encoding
<
...
* Ignoring the response-body
{ [43 bytes data]
* Connection #0 to host localhost left intact
* Maximum (50) redirects followed
* Closing connection 0
bug query-frontend help wanted
Source
picture gburek-fastly

Most helpful comment

I think I can fix it. Is it alright if I work on this?

picture aribalam on 20 Sep 2020
👍2

All 4 comments

I think this issue is somewhat similar to #3043. I fixed this for the new UI but I think the same thing is happening in the case of having a path prefix. This comment will give more idea to anyone who wants to work on this.

prmsrswt picture prmsrswt on 20 Sep 2020

I think I can fix it. Is it alright if I work on this?

aribalam picture aribalam on 20 Sep 2020
👍2

@aribalam Please go ahead!

yeya24 picture yeya24 on 20 Sep 2020

@yeya24 @prmsrswt As correctly pointed out in this comment, the trailing slashes is getting removed each time from the request URL, resulting in an infinite loop on visiting the bare path prefix.
On careful observation, I guess this is caused by a handler of the Cortex frontend that the query frontend is using. That is why no such issue is found in the querier.

I decided to separately handle this URL in the querier and redirect to the final page.
So a query to \thanos would redirect to \thanos\graph directly, instead of \thanos -> \thanos\ -> \thanos\graph.

I will be creating a PR for this.
Please correct me if I am wrong and let me your thoughts on this!

aribalam picture aribalam on 22 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bwplotka picture bwplotka  路  3Comments
jdfalk picture jdfalk  路  3Comments
abursavich picture abursavich  路  4Comments
MrD2 picture MrD2  路  3Comments
barryib picture barryib  路  4Comments