Envoy: How can I config envoy proxy to failover when some upstream being unreachable

Created on 19 Sep 2019 · 9Comments · Source: envoyproxy/envoy

Hi,
I'm new to this, so please bear with me.

What I am trying to do is use envoy as a sidecar proxy for grpc, I have following configuration, basically route grpc client to two service :

admin:
  access_log_path: "/tmp/admin_access.log"
  address:
    socket_address:
      address: "10.19.17.188"
      port_value: 12000
static_resources:
  listeners:
    -
      name: "grpc-listener"
      address:
        socket_address:
          address: "10.19.17.188"
          port_value: 12001
      filter_chains:
        -
          filters:
            -
              name: "envoy.http_connection_manager"
              config:
                stat_prefix: "ingress"
                codec_type: "AUTO"
                route_config:
                  name: "grpc-route"
                  virtual_hosts:
                    -
                      name: "grpc-route"
                      domains:
                        - "*"
                      routes:
                        -
                          match:
                            prefix: "/"
                          route:
                            cluster: "grpc-service"
                http_filters:
                  -
                    name: "envoy.router"

  clusters:
      -
        name: "grpc-service"
        connect_timeout: "0.25s"
        type: "static"
        lb_policy: "ROUND_ROBIN"
        http2_protocol_options: {}
        hosts:
          -
            socket_address:
              address: "10.19.17.188"
              port_value: 12011
          -
            socket_address:
              address: "10.19.17.188"
              port_value: 12012

This worked fine, when both service is up and running, but when I stop one service, client call would fail with following log:

[2019-09-18 12:29:02.380][798][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[2019-09-18 12:29:02.380][798][debug][http] [source/common/http/conn_manager_impl.cc:1111] [C285][S679184262732628339] request end stream
[2019-09-18 12:29:02.380][798][debug][connection] [source/common/network/connection_impl.cc:561] [C286] delayed connection error: 111
[2019-09-18 12:29:02.380][798][debug][connection] [source/common/network/connection_impl.cc:190] [C286] closing socket: 0
[2019-09-18 12:29:02.380][798][debug][client] [source/common/http/codec_client.cc:82] [C286] disconnect. resetting 0 pending requests
[2019-09-18 12:29:02.380][798][debug][pool] [source/common/http/http2/conn_pool.cc:149] [C286] client disconnected
[2019-09-18 12:29:02.380][798][debug][router] [source/common/router/router.cc:868] [C285][S679184262732628339] upstream reset: reset reason connection failure
[2019-09-18 12:29:02.380][798][debug][http] [source/common/http/conn_manager_impl.cc:1186] [C285][S679184262732628339] Sending local reply with details upstream_reset_before_response_started{connection failure}
[2019-09-18 12:29:02.380][798][debug][http] [source/common/http/conn_manager_impl.cc:1378] [C285][S679184262732628339] encoding headers via codec (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure'
'date', 'Wed, 18 Sep 2019 12:29:02 GMT'
'server', 'envoy'

So, How can I config envoy to deal with this scenario?

question

Source