Envoy: Taking a host out of rotation in a custom healthchecker

[canselcik]

Hi,

I am working on a custom healthchecker and I noticed

host_->setActiveHealthFailureType(Envoy::Upstream::Host::ActiveHealthFailureType::UNHEALTHY);
handleFailure(envoy::data::core::v2alpha::HealthCheckFailureType::ACTIVE);

on its own wasn't sufficient in taking the host out of rotation. The host still receives traffic after the code block above is executed.

Am I missing anything? Should I be calling host_->used(false) as well?

Thanks!

[mattklein123]

You need to ultimately call handleFailure(...)

[canselcik]

Thanks for the quick response as usual @mattklein123. I am already calling handleFailure(...) with HealthCheckFailureType::ACTIVE and even seeing the following but the host is still kept in rotation for the cluster:

[2019-05-20 15:14:10.253][268145][info][hc] [ns-platform/healthcheckers/custom.h:20] CustomHC active healthcheck -- onResponseComplete for origin 10.200.XXX.YYY:18335
[2019-05-20 15:14:10.253][268145][info][hc] [ns-platform/healthcheckers/custom.h:60] CustomHC active healthcheckResult -- response_code: 404
[2019-05-20 15:14:10.253][268145][info][hc] [ns-platform/healthcheckers/custom.h:31] CustomHC active healthcheck -- onResponseComplete failed
{"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"10.200.XXX.YYY","resolver_name":"","ipv4_compat":false,"port_value":18335}},"cluster_name":"customservice/somecluster","health_check_failure_event":{"failure_type":"ACTIVE","first_check":true},"timestamp":"2019-05-20T22:14:10.253Z"}

However Envoy is still routing traffic to these hosts as far as I can tell. Even though their health_flags are showing hc response=404 health_flags=/failed_active_hc.

What I am wondering is if there is some sort of config parameter I am missing that's making Envoy route to origins with health_flags == /failed_active_hc?

Thanks again.

[mattklein123]

Most likely panic mode: https://www.envoyproxy.io/docs/envoy/latest/faq/lb_panic_threshold

[canselcik]

Yup, good call, definitely seems to be the case. Thanks @mattklein123.

On a slightly unrelated note, any hints on how to use StreamInfoImpl to make a POST request with a body? I am able to get it to make a POST request instead of a GET but I have been unable to have it send a request body.

What I have, inside onInterval() of my class inheriting ActiveHealthCheckSession is this:

// This wants a decoder so I can't inject it in `decodeData(..)` either
Envoy::Http::StreamEncoder* request_encoder = &client_->newStream(*this);
request_encoder->getStream().addCallbacks(*this);

const Envoy::Http::HeaderValues& hn = Envoy::Http::Headers::get();
Envoy::Http::HeaderMapImpl request_headers {
  { hn.Method, "POST" },
  { hn.Host, host_->address()->asString() },
  { hn.Path, "/" },
  { hn.UserAgent, "someUseragent" },
  { hn.Authorization, "someAuthorization" }
};

Envoy::Router::FilterUtility::setUpstreamScheme(request_headers, *parent_.cluster_.info());
Envoy::StreamInfo::StreamInfoImpl stream_info(protocol_, parent_.dispatcher_.timeSource());
stream_info.setDownstreamLocalAddress(local_address_);
stream_info.setDownstreamRemoteAddress(host_->address());
stream_info.onUpstreamHostSelected(host_);
request_encoder->encodeHeaders(request_headers, true);

Writing this made me realize there might be a Envoy::Http::StreamDecoder I could use to get the state machine going instead of the StreamEncoder however client_->newStream(..) still expects something that implements the StreamDecoder interface.

Perhaps the UpstreamRequest interface?

[canselcik]

Found the answer to my question. For people who may end up on this thread looking into something similar, I am including the answer here:

  Envoy::Http::StreamEncoder& encoder = client.newStream(*response);
  encoder.getStream().addCallbacks(*response);

  Envoy::Http::HeaderMapImpl headers;
  headers.insertMethod().value(method);
  headers.insertPath().value(url);
  headers.insertHost().value(host);
  headers.insertScheme().value(Envoy::Http::Headers::get().SchemeValues.Http);    
  headers.insertContentType().value(content_type);

  encoder.encodeHeaders(headers, false);
  Envoy::Buffer::OwnedImpl body_buffer(payload);
  encoder.encodeData(body_buffer, true);