Beats: [Heartbeat] HTTP 302 redirect behavior

[andrewkroh]

I'm monitoring an HTTP endpoint that returns a 302 redirect to send clients to the HTTPS site. Heartbeat is making 10 requests for each check instead of just one. It's a lot more HTTP requests than I expected to see.

Ideally for this use case Heartbeat would not follow redirects. I didn't find anything in the documentation that describes Heartbeat's redirect following behavior. But the Go HTTP client's default behavior is:

// If CheckRedirect is nil, the Client uses its default policy,
// which is to stop after 10 consecutive requests.

• Version: Heartbeat 6.6.0
• Operating System: Centos 6
• Steps to Reproduce: Here's the config that I'm using:

heartbeat.monitors:
    - 
      check.response.status: 302
      tags: 
        - internal
        - redirect
      schedule: "@every 30s"
      type: http
      mode: all
      urls: 
        - "http://va.crowbird.com:80"

The DNS response returns a single private IP address.

$ dig va.crowbird.com
;; ANSWER SECTION:
va.crowbird.com.    3600    IN  A   10.x.y.z

This screenshot shows two consecutive checks from Heartbeat as captured by Packetbeat. Each row represents one HTTP requests made by Heartbeat.

[elasticmachine]

Pinging @elastic/uptime

[andrewvc]

@andrewkroh I agree that heartbeat should not follow redirects. Thanks for finding this!

[plog17]

@andrewkroh I agree that heartbeat should not follow redirects. Thanks for finding this!

When this will be addressed, could it be made configurable ?
I believe both use cases are useful and deciding if or not redirect should be followed should be configurable.

[petericebear]

Same thing over here. We have a https://link.to/12345 which redirects to a https://verylongdomainwithmanychars.com/share/12345.

Wanted to check the redirector if there is a 302, and the SSL certificate status.

[fabionitto]

I'm using Heartbeat 7.2.0 and the behaviour when trying to check for 302 response is that not only it follows the redirect automatically, but apparently it does not follow locations from different domains, for example:
The following monitor

type: http
schedule: '@every 5s'
urls: ["http://domain.com:80/redirect"]
check.response.status: 302

the response to http://domain.com/redirect is a 302 with Location: https://otherdomain.com/destination

Thus, the monitor always receive a 503 error, as it only consider the location path and tries to reach domain.com/destination which does not exists.

[andrewvc]

Thanks for the report @fabionitto that's definitely a bug, we'll try and take a look at it relatively soon.

[fabionitto]

@andrewkroh I agree that heartbeat should not follow redirects. Thanks for finding this!

When this will be addressed, could it be made configurable ?
I believe both use cases are useful and deciding if or not redirect should be followed should be configurable.

Also, I managed to check the code and found the max_redirects config parameter, which defines how many redirects Heartbeat will follow. You can set it with 0 redirects and then it will receive the 302 HTTP code as expected.

If this config is fully functional I think it could be added to documentation, which apparently is missing.
Following is an example config:

type: http
schedule: '@every 5s'
max_redirects: 0
urls: ["http://domain.com:80/redirect"]
check.response.status: 302

[andrewvc]

Thanks for the patch @fabionitto . This is actually a harder problem to solve than it would appear. The main reason being the presence of mode: all. This will attempt to hit all IPs listed for the given domain with separate checks. In the case of a redirect to another domain it appears that we use our previous DNS lookups to stay pinned to the IP we found hence not switching domains.

One possible solution would be to only allow redirects with mode: any. However, then we have to think about what goes in monitor.ip. Do we leave it blank? Just use the first IP? Make it a list of all IPs?

I think the stronger approach would be to spawn an additional check for each redirect, yielding an additional document. This is actually easy enough in heartbeat since we use continuation passing style internally, however, from a UX/UI side this gets tricky to represent. However, it may be worth tackling.

@roncohen @katrin-freihofner I'd be curious to hear your thoughts on ^^

[katrin-freihofner]

🤔I agree, it should not automatically follow redirects. I think I would expect to get a 3XX redirect code. Additionally, it would be super nice to get a redirects to https://..... I'm not sure if this was helpful 🙈

[andrewvc]

Taking this all together I do think it makes sense to:

1. Deprecate redirect handling in 7.x
2. Remove it in 8.0.0

While there are valid use cases for redirects, I don't think we can handle them well now.

In the future this might be better handled by an enhanced transaction, or even synthetics mode where we can check a transaction across multiple URLs, but today this is just awkward.

[jsnider-mtu]

I'm seeing this issue as well with Heartbeat v7.3.0

[andrewvc]

I've created a tracking issue with a proposed fix here: https://github.com/elastic/beats/issues/13146 . Would appreciate input from those in this thread on the approach!

[andrewvc]

Fixed via https://github.com/elastic/beats/pull/14125 , release targeting 7.6