Runtime: HTTP2: Rare HPack decoding error on inbound headers frame

[eiriktsarpalis]

I came across the following error when stress testing HTTP2 against Kestrel:

System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.IO.IOException: The request was aborted.
---> System.Net.Http.HPack.HPackDecodingException: Dynamic table size update received after beginning of header block.
   at System.Net.Http.HPack.HPackDecoder.Decode(ReadOnlySpan`1 data, Boolean endHeaders, HeaderCallback onHeader, Object onHeaderState)
   at System.Net.Http.Http2Connection.ProcessHeadersFrame(FrameHeader frameHeader)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 93
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_2>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 201
--- End of stack trace from previous location where exception was thrown ---
   at HttpStress.StressClient.<>c__DisplayClass11_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/StressClient.cs:line 110

Relevant code

It occurred when sending a GET request which echoes a large number of randomly generated headers. Issue seems to be extremely rare, I've come across it only once so far.

cc @scalablecory

[geoffkizer]

This looks like a Kestrel issue. Is there a Kestrel issue filed? Or is there some reason to believe this is actually an issue with HttpClient?

[davidsh]

@eiriktsarpalis What is the status of this? At this point, perhaps this doesn't block 3.0 release especially if it is rare to repro?

cc: @stephentoub

[eiriktsarpalis]

@davidsh I haven't seen this issue happen a second time. I think we can probably move to future.

[eiriktsarpalis]

This was a one-off error observed one month ago, and it hasn't reproduced since. Fair to close at this point?

[geoffkizer]

I'm fine to close.

[TechnikEmpire]

Re-open.
Here's everything you need to repro:

Win10 Pro x64 1909
.NET Core 3.1

2020-03-24 16:16:23.546 -04:00 [Error] Method: GET, RequestUri: 'https://f.vimeocdn.com/js_opt/_bundles/modal.bundle.min.js?052e70b7', Version: 2.0, Content: <null>, Headers:
{
  Accept: */*
  Accept-Encoding: gzip
  Accept-Encoding: deflate
  Accept-Encoding: br
  Accept-Language: en-US
  Accept-Language: en; q=0.9
  Host: f.vimeocdn.com
  Referer: https://vimeo.com/
  User-Agent: Mozilla/5.0
  User-Agent: (Windows NT 10.0; Win64; x64)
  User-Agent: AppleWebKit/537.36
  User-Agent: (KHTML, like Gecko)
  User-Agent: Chrome/80.0.3987.149
  User-Agent: Safari/537.36
  DNT: 1
  sec-fetch-dest: script
  sec-fetch-site: cross-site
  sec-fetch-mode: no-cors
}    Handle::MY_REDACTED_SOURCE_PATH.cs() #463
2020-03-24 16:16:23.547 -04:00 [Error] An error occurred while sending the request.   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at CitadelService.Proxy.Net.Handlers.MY_REDACTED_SOURCE_PATH.Handle(HttpContext context) in [MY_REDACTED_SOURCE_PATH].cs:line 459The request was aborted.   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)Exception of type 'System.Net.Http.HPack.HPackDecodingException' was thrown.   at System.Net.Http.HPack.HPackDecoder.GetHeader(Int32 index)
   at System.Net.Http.HPack.HPackDecoder.OnIndexedHeaderField(Int32 index, HeaderCallback onHeader, Object onHeaderState)
   at System.Net.Http.HPack.HPackDecoder.Decode(ReadOnlySpan`1 data, Boolean endHeaders, HeaderCallback onHeader, Object onHeaderState)
   at System.Net.Http.Http2Connection.ProcessHeadersFrame(FrameHeader frameHeader)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
     From Handle::[MY_REDACTED_SOURCE_PATH].cs() #464

[karelz]

@TechnikEmpire I don't understand why you posted your last unrelated comment. Maybe if we keep discussions technical, that would be best ...

In general, we do not monitor closed issues. Luckily we noticed this one.
Can you please create a new issue?
Does it reproduce for you in a standalone HelloWorld app sending a single request to this URL?

BTW: Why do you think your exception / problem (which seems to be 100% repro, right?) is the same as the one in this issue. I don't see much similarity in callstacks. Am I missing something?

[richlander]

I posted it because I feel gross contributing to a project that is now governed by a policy ...

Then feel free not to contribute. The policy isn't going to change.

[terrajobst]

@TechnikEmpire this isn’t the place to discuss the Code of Conduct (CoC). Please keep issues in this repository focused on aspects concerning the .NET runtime & base class libraries. Furthermore, we don’t appreciate your approach of communicating with us regarding a closed issue. It’s fine to disagree with our design choices and trade off decisions. However, just because you disagree with our CoC doesn’t mean we’re going to accept your behavior. We’ve blocked your account for one week. If this behavior persists, we’ll block your account permanently.