Go: x/net/http2: ReadIdleTimeout/PingTimeout not exposed if configured through http package

[JensErat]

What version of Go are you using (go version)?

$ go version
go version go1.14.4 linux/amd64

Does this issue reproduce with the latest release?

Yes, following code on golang/net master branch.

What operating system and processor architecture are you using (go env)?

not relevant

What did you do?

golang/net#55 brought a new feature in golang's http2 implementation that allows early detection of broken network connections. Setting this seems to resolve potentially several rare issues we're facing in production every now and then.

Now, most clients use the (I think also recommended?) way to instantiate an http.Transport, which is then "upgraded" to http2 capabilities. For example, in prometheus/common:

        var rt http.RoundTripper = &http.Transport{
                  // ...
        }
        err := http2.ConfigureTransport(rt.(*http.Transport))

or they set Transport.ForceAttemptHTTP2 = true.

Both ways entirely hide the http2 transport, and I do not see a way to configure http2.Transport.ReadIdleTimeout/http2.Transport.PingTimeout.

This also seems to affect k8s.io/apimachinery/pkg/util/net/http.go, where this golang feature is best bet so far to resolve kubernetes/kubernetes#87615 (it seems to me golang/net#55 was specifically build because of the Kubernetes issue). golang/net#74 proposes an interface to set those values and would resolve this issue.

What did you expect to see?

A possibility to configure http2.Transport from http.Transport where needed, for example by passing in an http2.Transport into the ConfigureTransport function?

What did you see instead?

No way to configure http2.Transport settings.

[andybons]

@bradfitz @neild

[JensErat]

And maybe @caesarxuchao if I can notify him at all this way.

[networkimprov]

@fraenkel is the go-to guy for HTTP2.

[caesarxuchao]

I created https://github.com/golang/net/pull/74 but it was too late to get into 1.15.

[caesarxuchao]

FWIW, in the test I added in https://github.com/golang/net/pull/74, I used the unsafe package to configure the two fields. Obviously, it's not the right way, but can be the last resort.

[JensErat]

In prometheus/common, x/net/http2 is vendored so I could easily "hack" the field in there. At least we have somewhat good confidence it resolves our Prometheus issues now. I indeed missed golang/net#74 when searching for a solution. Since at least Prometheus seems to actually vendor the code, we might be fine over there as soon as your PR is merged (they pin some very recent master commit, anyway).

[roidelapluie]

Do you know if using ForceAttemptHTTP2 (go 1.13+) instead of http2.ConfigureTransport would solve that?

[povsister]

@roidelapluie
Noop. ForceAttemptHTTP2 only attempts to upgrade http1.1 to h2 and it's default enabled if my memory is right.
In current implement of h2. The http2 transport never dials tcp connection. It always let http1.1 dial and then upgrade it to h2.
The upgrade procedure is done by http2.ConfigureTransport.

Unfortunately, the h2 transport is not exposed by net/http/Transport which is the default transport for http client.
https://github.com/golang/go/blob/fa98f46741f818913a8c11b877520a548715131f/src/net/http/transport.go#L262-L266

So basically, it's impossible to directly manipulate the h2 transport out of net/http package.
It also means you can not manually enable the h2 connection healthcheck in your application. If you really wish to, you need "hack" the Transport like @caesarxuchao does so.

@JensErat is right. Currently no options is provided for upgrading h1 to h2. We'd better make it available ASAP. It really happens rarely but leads to disaster if you suddenly knock into it. See https://github.com/golang/go/issues/39750

https://github.com/golang/go/blob/fa98f46741f818913a8c11b877520a548715131f/src/net/http/h2_bundle.go#L6632-L6645

[JensErat]

We can confirm this resolves our Prometheus and Kubernetes issues (and thus likely the issues of lots of others, some might not even be aware of this).

@brian-brazil from Prometheus already proposed the health check should be default anyway:

My point is more that the developer shouldn't even need to configure this, it should be enabled by default.

And I pretty much agree with him. This is something most applications will mess up. Lots of admin and developer days lost in debugging a hard to trace down issue. If I have long-lasting connections, those should be robust against network failures.

Could this be considered a breaking change or just a fix? Issues with broken servers that cannot handle whatever http2 calls are sent to run the ping?

A golang debug variable to set the value might also help in either case (force-enable or force-disable).

[gopherbot]

Change https://golang.org/cl/236498 mentions this issue: http2: add configuration options when constructing http2 transport

[gopherbot]

Change https://golang.org/cl/264017 mentions this issue: x/net/http2: add ConfigureTransports