Envoy: Major Blocker: Envoy causes java 11+ http2 connection failure in Istio

[jingnanzhou]

Envoy seems to cause a major issue for Java 11+ HTTPClient connection. Here is the bug found in Istio ( https://github.com/istio/istio/issues/16391 ). Is anyone aware of the issue

Bug description

Java 11+ core HttpClient optionally supports HTTP/2, and as per official HTTP/2 spec rfc7540, it sends optional Upgrade: h2c header, which is purely indicative and can/should be ignored and treated as any other X- header if the target doesn't support HTTP/2. The HTTP/2 spec shows clearly on how the header can be ignored and normal HTTP/1.1 response can be returned.

In Istio, even though we explicitly name the port as http (and NOT http2), and the http/2 spec explicitly states these headers as optional, it treats these optional headers as mandatory, and returns a confusing HTTP 403 Forbidden.

This is not an exotic use case. This basically breaks all Java 11 applications and beyond, and most Java 9 applications that use spring-boot, the de-facto java web framework, since spring-boot uses the exact same HTTP/2 functionality under the hoods from Java 9 onwards.

It may break other libraries in other languages/platforms which normally implement the HTTP/2 spec to send these optional headers. It has been easily reproduced with curl --http2. I can submit test cases for most other language libraries.

Why is this called "all java 11+ and partially 9+applications"? Because it breaks the core language sdk, and all applications who are making a http call are impacted. Those who don't, don't know they're impacted and just sitting on somewhere in the dependency tree (like an oauth client or something) to break. Not withstanding all language libraries and servers that implement http/2 spec as per the optional headers.

[mattklein123]

See https://github.com/envoyproxy/envoy/issues/7161#issuecomment-523018614 for how to pass through h2c upgrade. Doc updates appreciated for this case.

[rshriram]

@mattklein123 that comment basically says that people need to write custom filters to make envoy work with standard java sdks. Passing h2c to filter chains and then stripping it out in some special filter is same as dropping h2 altogether as at cluster level, we will still do h1 to the upstream app.

[mattklein123]

So what is the request here? To ignore h2c and not upgrade and stick to HTTP/1.1?

[mattklein123]

We already have https://github.com/envoyproxy/envoy/issues/7161 fully tracking h2c upgrade.

[rshriram]

https://github.com/rdsubhas/java-istio/issues/1

[mattklein123]

Correct, Envoy does not support h2c upgrades. This is already tracked in https://github.com/envoyproxy/envoy/issues/7973. If there is some different request here let me know.

[snowp]

This basically breaks all Java 11 applications and beyond

This seems like a bit of an exaggeration, we've been happily using Envoy with Java 9/10/11 for months now, using a Jetty/Netty/Okhttp based stack.

[lizan]

@rshriram I believe you only need a custom config, rather than writing a custom filter. OTOH, can't java client use h2c with prior knowledge (i.e. what curl --http2-prior-knowledge does)?

[rshriram]

I have no idea. Per @Stono, seems the breakage is caused by the default stdlibs in java supporting inplace upgrades (just like the websockets).

[mattklein123]

This will be worked around via https://github.com/envoyproxy/envoy/pull/7981 and we are tracking actual h2c support in https://github.com/envoyproxy/envoy/issues/7973

[rdsubhas]

@rshriram @mattklein123 The issue raised by @Stono was wrong and has been closed as invalid.

It's not about "not sending mandatory headers" like the issue seems to nudge. The implementation is broken even if you send the mandatory headers and follow the spec. So this is not a case of feature request or enhancement. It's a spec break.

https://github.com/rdsubhas/java-istio/issues/1#issuecomment-523806004

[lavignes]

Just wanted to chime in on here and mention that we've verified that App Mesh is also affected by this. We're looking forward to #7981 getting merged.