Httpx: H2 StreamClosed error while requesting duckduckgo.com

[yeraydiazdiaz]

Running the following script on the current master:

import asyncio
import httpx

async def main():
    cl = httpx.AsyncClient()
    resp = await cl.get("https://duckduckgo.com")
    print(resp)

if __name__ == "__main__":
    asyncio.run(main())

Errors with the following traceback:

Traceback (most recent call last):
  File ".dev/middlewares.py", line 14, in <module>
    asyncio.run(main())
  File "/Users/yeray/.pyenv/versions/3.7.3/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Users/yeray/.pyenv/versions/3.7.3/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File ".dev/middlewares.py", line 8, in main
    resp = await cl.get("https://duckduckgo.com")
  File "/Users/yeray/code/personal/_forks/httpx/httpx/client.py", line 327, in get
    timeout=timeout,
  File "/Users/yeray/code/personal/_forks/httpx/httpx/client.py", line 560, in request
    timeout=timeout,
  File "/Users/yeray/code/personal/_forks/httpx/httpx/client.py", line 155, in send
    allow_redirects=allow_redirects,
  File "/Users/yeray/code/personal/_forks/httpx/httpx/client.py", line 188, in send_handling_redirects
    request, verify=verify, cert=cert, timeout=timeout
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/connection_pool.py", line 116, in send
    raise exc
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/connection_pool.py", line 111, in send
    request, verify=verify, cert=cert, timeout=timeout
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/connection.py", line 51, in send
    response = await self.h2_connection.send(request, timeout=timeout)
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/http2.py", line 48, in send
    status_code, headers = await self.receive_response(stream_id, timeout)
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/http2.py", line 122, in receive_response
    event = await self.receive_event(stream_id, timeout)
  File "/Users/yeray/code/personal/_forks/httpx/httpx/dispatch/http2.py", line 155, in receive_event
    events = self.h2_state.receive_data(data)
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h2/connection.py", line 1463, in receive_data
    events.extend(self._receive_frame(frame))
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h2/connection.py", line 1486, in _receive_frame
    frames, events = self._frame_dispatch_table[frame.__class__](frame)
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h2/connection.py", line 1702, in _receive_window_update_frame
    frame.window_increment
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h2/stream.py", line 1126, in receive_window_update
    StreamInputs.RECV_WINDOW_UPDATE
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h2/stream.py", line 129, in process_input
    return func(self, previous_state)
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h2/stream.py", line 397, in window_on_closed_stream
    return self.recv_on_closed_stream(previous_state)
  File "/Users/yeray/.pyenv/versions/httpx/lib/python3.7/site-packages/h2/stream.py", line 336, in recv_on_closed_stream
    raise StreamClosedError(self.stream_id)
h2.exceptions.StreamClosedError: 1

Anyone else getting this error?

[sethmlarson]

I can reproduce that error. We need to think about integrating debug logging into some of our classes otherwise it's impossible to figure out what's wrong as a user.

[sethmlarson]

(Even with debug logging it isn't obvious what the issue is here, you get a big blob back and then h2 complains about the stream being closed)

[sethmlarson]

So, it looks like DuckDuckGo's HTTP/2 implementation is sending a Window Update frame on stream 1 after itself closes stream 1 and so H2 is freaking out over it. This is the order h2 is seeing the events after a request is sent:

SettingsFrame(Stream: 0; Flags: ACK): 
HeadersFrame(Stream: 1; Flags: END_HEADERS): 20887684aa6355e76196...
DataFrame(Stream: 1; Flags: END_STREAM): d1a8a800203eeda7f51d...
WindowUpdateFrame(Stream: 1; Flags: None): 7ffeffff

I don't know if this is wrong or not, seems strange to me. cc @pgjones @njsmith

[njsmith]

I don't know anything about this, I think you'd have to read the RFC or ask @lukasa...

[Lukasa]

This isn’t technically wrong, but h2 doesn’t love it. Of course, depending on the timings that END_STREAM frame may have been in flight when DDG generated the WINDOW_UPDATE. We should probably patch h2 to be more tolerant in this case.

[sethmlarson]

I've opened up https://github.com/python-hyper/hyper-h2/issues/1196 related to this issue.

[vmagamedov]

This is already fixed in h2, waiting for a new release.

[sethmlarson]

Verified that the latest master for h2 fixes this issue.

[pgjones]

I've released h2 3.1.1, hopefully this is all good now.

[sethmlarson]

Verified that this is working in h2==3.1.1. Thanks to the Hyper team and @pgjones! :heart: