Grpc-web: upstream connect error or disconnect/reset before headers. reset reason: connection termination

[wkmian]

The client (js, c++) does not operate for a long time and will prompt "upstream connect error or disconnect/reset before headers. reset reason: connection termination". And the request again is correct.

I don't know how to deal with this problem.

[weilip1803]

Did you set max_grpc_timeout to 0 seconds in envoy?

          - match: { prefix: "/asda/" }
            route: { cluster: asda, prefix_rewrite: "/", max_grpc_timeout: 0s }

[wkmian]

Is c++ and java also the problem? I try it！

[wkmian]

Did you set max_grpc_timeout to 0 seconds in envoy?

          - match: { prefix: "/asda/" }
            route: { cluster: asda, prefix_rewrite: "/", max_grpc_timeout: 0s }

This is my configuration. But this problem still exists.
The default is 0, now I changed to 2s

routes:
    - match: { prefix: "/" }
    route:
        cluster: greeter_service
        max_grpc_timeout: 2s

[weilip1803]

I dont think the default is 0s. I think 0s means no timeout. I am running against with c++/java/python and nodejs just fine.

Hmmm I used to have this problem when I did not set max_grpc_timeout to 0s. You can checkout this thread https://github.com/grpc/grpc-web/issues/361 . Hope it helps you. Good luck

[wkmian]

Thank you ! Let me find something else.

[MDBox]

We faced the same error. The reason for us was because tcp_keepalive was set too high for our upstream service. We changed keepalive_time to 300 seconds and the problem went away.

It makes sense in our case because we have Envoy pointing to a network load balancer (aws) which has a 350s idle timeout.

We added this to our envoy config.

clusters:
    - name: grpc-service
      connect_timeout: 0.25s
      http2_protocol_options: {}
      upstream_connection_options:
        tcp_keepalive:
          keepalive_time: 300

AWS Source: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html

Elastic Load Balancing sets the idle timeout value to 350 seconds. You cannot modify this value. The targets of a TCP listener can use TCP keepalive packets to reset the idle timeout. TCP keepalive packets are not supported for the targets of a TLS listener.

Envoy Default: https://www.envoyproxy.io/docs/envoy/v1.10.0/api-v2/api/v2/core/address.proto#envoy-api-msg-core-tcpkeepalive

The number of seconds a connection needs to be idle before keep-alive probes start being sent. Default is to use the OS level configuration (unless overridden, Linux defaults to 7200s (ie 2 hours.)

Hope it helps

[wkmian]

We faced the same error. The reason for us was because tcp_keepalive was set too high for our upstream service. We changed keepalive_time to 300 seconds and the problem went away.

It makes sense in our case because we have Envoy pointing to a network load balancer (aws) which has a 350s idle timeout.

We added this to our envoy config.

clusters:
    - name: grpc-service
      connect_timeout: 0.25s
      http2_protocol_options: {}
      upstream_connection_options:
        tcp_keepalive:
          keepalive_time: 300

AWS Source: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html

Elastic Load Balancing sets the idle timeout value to 350 seconds. You cannot modify this value. The targets of a TCP listener can use TCP keepalive packets to reset the idle timeout. TCP keepalive packets are not supported for the targets of a TLS listener.

Envoy Default: https://www.envoyproxy.io/docs/envoy/v1.10.0/api-v2/api/v2/core/address.proto#envoy-api-msg-core-tcpkeepalive

The number of seconds a connection needs to be idle before keep-alive probes start being sent. Default is to use the OS level configuration (unless overridden, Linux defaults to 7200s (ie 2 hours.)

Hope it helps

Thank you ！ I think it is very helpful ！