Envoy: Custom upstrem/backend response

Created on 29 Aug 2020 · 11Comments · Source: envoyproxy/envoy

I want to rewrite the 4xx/5xx returned by the upstrem/backend service.
I see that envoy 1.15.0 has been able to rewrite the error of envoy itself through local_reply_config, but it seems that it cannot rewrite the error returned by upstrem/backend. The function of local_reply_config is very good and can replace haproxy as the front end lb. It would be even better if could rewrite upstrem/backend.

I don’t know if my configuration is wrong, my configuration：

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: local-reply
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER # http connection manager is a filter in Envoy
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          local_reply_config:
            mappers:
            - filter:
                status_code_filter:
                  comparison:
                    op: EQ
                    value:
                      default_value: 503
                      runtime_key: key_b
              status_code: 200
              body:
                inline_string: "no health server"
              body_format_override:
                json_format:
                  status: "503"
                  message: "%LOCAL_REPLY_BODY%"
                  request_path: "%REQ(:path)%"

[root@k8s-20 ~]# curl -i http://test.domain.com/503/test/path                                                                                            
HTTP/1.1 503 Service Unavailable
server: nginx/1.15.6
date: Sat, 29 Aug 2020 03:51:36 GMT
content-type: text/html
content-length: 203
x-request-id: 3c4a04cd-c4d3-4819-af76-256b6c905a0b

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>

question

Source

tony-liuliu

👀1

Most helpful comment

Yes, I think this feature is very useful because our back-end services are provided by different languages, such as nginx, tomcat, nodejs, go. If you do not manually compile and configure them, their server information will expose it when an error is encountered, which is prone to insecure factors. Using proxy_intercept_errors like nginx to unify redirect errors seems very suitable for this scenario. This way we don't have to modify error redirects such as http 404 for each backend language. I hope that envoy can consider such needs in future plans.

tony-liuliu on 11 Sep 2020

👍8

All 11 comments

We don't have generic rewrite functionality currently. This could be implemented by a custom filter or Lua potentially.

mattklein123 on 1 Sep 2020

@mattklein123 Thank you very much, I will try to use custom filter. Is there a similar custom filter example.

tony-liuliu on 1 Sep 2020

@tony-liuliu do you think you can make it available upstream? Thinking to have a way to modify the proxied response (with non-2xx status), do you think it makes sense to have an error_page "filter" similar to what NGINX has? (in NGINX, we need to turn on proxy_intercept_errors to let non-2xx proxied response to be handled by the proxy's error_page config).

I can see it can also be implemented in Lua (with some thinking on how to not complicate the "state management" there). Relevant: https://github.com/envoyproxy/envoy/issues/5998.

dio on 9 Sep 2020

tony-liuliu on 11 Sep 2020

👍8

@tony-liuliu if you can help us with a design doc, I think I can help you with the implementation.

cc. @mattklein123

dio on 11 Sep 2020

I think it would be great to have a custom response encoder filter that could do something like this.

Note also this would go really well with the work outlined here to allow for response matching: https://github.com/envoyproxy/envoy/issues/12968. cc @snowp

mattklein123 on 11 Sep 2020

I tried to do something similar with an AssemblyScript WASM filter and failed.

My filter's onResponseHeaders() could see the 4xx/5xx returned by the backend. I tried calling send_local_response() from within onResponseHeaders() but by then it was too late.

I notice FilterHeadersStatus .ContinueAndDontEndStream was recently added which might solve my problem but that return isn't yet available in the Envoy version I am using nor in the Solo.io AssemblyScript runtime.

esnible on 15 Sep 2020

Another ref: https://www.haproxy.com/blog/serve-dynamic-custom-error-pages-with-haproxy/

dio on 29 Sep 2020

@dio @mattklein123 Is this sufficient for a design doc? https://docs.google.com/document/d/1Bgi0agddua9LyBH3NnGBuDPcYJxoZwZLnL1rBcKFgZI

ceastman-ibm on 13 Oct 2020

This behavior should be implemented as a filter. It's not clear to me that is what is being proposed? Is it?

mattklein123 on 13 Oct 2020

I ended up implementing this as an HTTP filter before I knew this issue (or any design documents/requirements) existed. For that reason, my approach may be different than what's being discussed here but I'm very much open to feedback. Please see https://github.com/envoyproxy/envoy/pull/14221 ( cc @mattklein123 / @tony-liuliu )