Httpx: SyncHTTP11Connection ImportError on Python 3.9 when using ThreadPoolExecutor

Created on 19 Nov 2020 · 6Comments · Source: encode/httpx

Checklist

[x] The bug is reproducible against the latest release and/or master.
[x] There are no similar issues or pull requests to fix it yet.

Describe the bug

When using httpx with ThreadPoolExecutor on Python 3.9 I get an exception about the import error / circular import of SyncHTTP11Connection.

I can't reproduce it on Python 3.8.

To reproduce

from concurrent.futures import ThreadPoolExecutor, as_completed

import httpx


_MAX_WORKERS = 4
_TASKS = 10

_HTTPBIN_BASE_URL = "https://httpbin.org/"


def do_post(client: httpx.Client):
    response = client.post("post")
    return response


def main():
    client = httpx.Client(timeout=5.0, base_url=_HTTPBIN_BASE_URL)

    with ThreadPoolExecutor(max_workers=_MAX_WORKERS) as executor:
        futures = [executor.submit(do_post, client) for _ in range(_TASKS)]

        for future in as_completed(futures):
            response = future.result()
            print(f"{response.status_code=}")


if __name__ == "__main__":
    main()

With _MAX_WORKERS=4 I get the error, but sometimes, when I change its value it works fine. Sometimes.

Expected behavior

It should not throw an exception regarding circular imports when using ThreadPoolExecutor. :-)

Actual behavior

Well, it throws it. :D

Debugging material

Traceback with HTTPX_LOG_LEVEL=trace

$ HTTPX_LOG_LEVEL=trace python main.py 
TRACE [2020-11-19 18:15:10] httpx._config - load_ssl_context verify=True cert=None trust_env=True http2=False
TRACE [2020-11-19 18:15:10] httpx._config - load_verify_locations cafile=/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/certifi/cacert.pem
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - create_connection socket=<httpcore._backends.sync.SyncSocketStream object at 0x7f8f90748520> http_version='HTTP/1.1'
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - create_connection socket=<httpcore._backends.sync.SyncSocketStream object at 0x7f8f8f6f87c0> http_version='HTTP/1.1'
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - remove from pool connection=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - create_connection socket=<httpcore._backends.sync.SyncSocketStream object at 0x7f8f8f703a00> http_version='HTTP/1.1'
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - removing connection from pool=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - remove from pool connection=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - removing connection from pool=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_data=Data(<0 bytes>)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool=<SyncHTTPConnection http_version=UNKNOWN state=0>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}
DEBUG [2020-11-19 18:15:10] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - receive_event=Data(<395 bytes>)
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - receive_event=EndOfMessage(headers=<Headers([])>)
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - reusing idle http11 connection=<SyncHTTPConnection http_version=HTTP/1.1 state=4>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - reuse connection=<SyncHTTPConnection http_version=HTTP/1.1 state=1>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_data=Data(<0 bytes>)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - create_connection socket=<httpcore._backends.sync.SyncSocketStream object at 0x7f8f8f6ed8e0> http_version='HTTP/1.1'
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_data=Data(<0 bytes>)
DEBUG [2020-11-19 18:15:10] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - receive_event=Data(<395 bytes>)
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - receive_event=EndOfMessage(headers=<Headers([])>)
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - reusing idle http11 connection=<SyncHTTPConnection http_version=HTTP/1.1 state=4>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - reuse connection=<SyncHTTPConnection http_version=HTTP/1.1 state=1>
TRACE [2020-11-19 18:15:10] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_data=Data(<0 bytes>)
DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=<Headers([])>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - reusing idle http11 connection=<SyncHTTPConnection http_version=HTTP/1.1 state=4>
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - reuse connection=<SyncHTTPConnection http_version=HTTP/1.1 state=1>
TRACE [2020-11-19 18:15:11] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_data=Data(<0 bytes>)
TRACE [2020-11-19 18:15:11] httpcore._sync.connection - create_connection socket=<httpcore._backends.sync.SyncSocketStream object at 0x7f8f8e675be0> http_version='HTTP/1.1'
TRACE [2020-11-19 18:15:11] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_data=Data(<0 bytes>)
TRACE [2020-11-19 18:15:11] httpcore._sync.connection - create_connection socket=<httpcore._backends.sync.SyncSocketStream object at 0x7f8f8e675df0> http_version='HTTP/1.1'
TRACE [2020-11-19 18:15:11] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_data=Data(<0 bytes>)
DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=<Headers([])>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443)
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - reusing idle http11 connection=<SyncHTTPConnection http_version=HTTP/1.1 state=4>
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - reuse connection=<SyncHTTPConnection http_version=HTTP/1.1 state=1>
TRACE [2020-11-19 18:15:11] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')]
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_data=Data(<0 bytes>)
DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=<Headers([])>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE
DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=<Headers([])>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE
DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=<Headers([])>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE
DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=<Headers([])>)
TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE
Traceback (most recent call last):
  File "/tmp/httpx-threadpool-demo/main.py", line 29, in <module>
    main()
  File "/tmp/httpx-threadpool-demo/main.py", line 24, in main
    response = future.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/tmp/httpx-threadpool-demo/main.py", line 13, in do_post
    response = client.post("post")
  File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 992, in post
    return self.request(
  File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 733, in request
    return self.send(
  File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 767, in send
    response = self._send_handling_auth(
  File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 805, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 837, in _send_handling_redirects
    response = self._send_single_request(request, timeout)
  File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 861, in _send_single_request
    (status_code, headers, stream, ext) = transport.request(
  File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 218, in request
    response = connection.request(
  File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 93, in request
    self._create_connection(self.socket)
  File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 153, in _create_connection
    from .http11 import SyncHTTP11Connection
ImportError: cannot import name 'SyncHTTP11Connection' from partially initialized module 'httpcore._sync.http11' (most likely due to a circular import) (/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py)
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - removing connection from pool=<SyncHTTPConnection http_version=HTTP/1.1 state=4>
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - removing connection from pool=<SyncHTTPConnection http_version=HTTP/1.1 state=4>
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - removing connection from pool=<SyncHTTPConnection http_version=HTTP/1.1 state=4>
TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - removing connection from pool=<SyncHTTPConnection http_version=HTTP/1.1 state=4>

It looks like it is actually doing requests, but fails afterwards?

Environment

OS: Ubuntu 20.04 on WSL2 on Windows 10
Python version: 3.9.0
HTTPX version: 0.16.1
HTTP proxy: no
Custom certificates: no

Additional context

I get the same error on the Windows version of Python 3.9 too.

bug htt1.1 httpcore

Source

Senpos

All 6 comments

Woah, that's umm, weird.

Does it reproduce on 3.9 only?
Does it reproduce consistently for each number of workers? (Eg does workers=4 consistently raise this exception, and does eg workers=3 consistently not raise it?)
Is there any bug report related to the threadpool executor on the Python bug tracker?

florimondmanca on 20 Nov 2020

Hi @florimondmanca!

Yes
Yes, with 4 workers; sometimes with 2-3 workers.
Not sure, sorry. I am not too familiar with it go get an idea if something is related to this

It works fine with requests.

from concurrent.futures import ThreadPoolExecutor, as_completed

import requests


_MAX_WORKERS = 4
_TASKS = 10

_HTTPBIN_BASE_URL = "https://httpbin.org/"


def do_post(client: requests.Session):
    response = client.post(_HTTPBIN_BASE_URL + "post")
    return response


def main():
    client = requests.Session()

    with ThreadPoolExecutor(max_workers=_MAX_WORKERS) as executor:
        futures = [executor.submit(do_post, client) for _ in range(_TASKS)]

        for future in as_completed(futures):
            response = future.result()
            print(f"{response.status_code=}")


if __name__ == "__main__":
    main()

Senpos on 20 Nov 2020

So, it's quite interesting.

I ran your script, and at first it worked fine. Now if I run it again, it fails consistently. If I turn on debug logging I can see it's making some of the requests (generally 7), doesn't show the output, then fails.

If I add a print("getting result") just before the call to future.result(), we can see it's hanging on it, and then we start to see requests terminating, then a crash before the future returns.

$ HTTPX_LOG_LEVEL=debug python debug/bug.py
getting result
DEBUG [2020-11-20 09:30:08] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:08] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:08] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:08] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
Traceback (most recent call last):
  File "/Users/florimond/Developer/python-projects/httpx/debug/bug.py", line 30, in <module>
    main()
  File "/Users/florimond/Developer/python-projects/httpx/debug/bug.py", line 25, in main
    response = future.result()
  File "/Users/florimond/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/Users/florimond/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/Users/florimond/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/florimond/Developer/python-projects/httpx/debug/bug.py", line 13, in do_post
    response = client.post("post")
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 992, in post
    return self.request(
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 733, in request
    return self.send(
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 767, in send
    response = self._send_handling_auth(
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 805, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 837, in _send_handling_redirects
    response = self._send_single_request(request, timeout)
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 861, in _send_single_request
    (status_code, headers, stream, ext) = transport.request(
  File "/Users/florimond/Developer/python-projects/httpx/venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 200, in request
    response = connection.request(
  File "/Users/florimond/Developer/python-projects/httpx/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 88, in request
    self._create_connection(self.socket)
  File "/Users/florimond/Developer/python-projects/httpx/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 136, in _create_connection
    from .http11 import SyncHTTP11Connection
ImportError: cannot import name 'SyncHTTP11Connection' from partially initialized module 'httpcore._sync.http11' (most likely due to a circular import) (/Users/florimond/Developer/python-projects/httpx/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py)

Crucially I've tested on 3.8, and I'm not able to reproduce there (even if on the same versions of HTTPX / HTTPCore). So I'm tempted to mark this as a 3.9 bug for now, though it may also be that something _changed_ in 3.9 that revealed a problem we might have with thread-safe execution.

florimondmanca on 20 Nov 2020

Ah, also a funny bit of info: this only reproduces on http11, not http2. Eg Client(..., http2=True) makes the bug go away. 🤔

florimondmanca on 20 Nov 2020

This bug likely comes from: https://bugs.python.org/issue35943 - Python 3.9 changed a bug which allowed imports to return partially initialized module.

This discussion ticket relates similar behavior seen with multiprocessing: https://discuss.python.org/t/differences-between-3-8-and-3-9-in-importing-module/5520/2

This discussion^ mentions that "multiprocessing has a lot of local imports, so multiprocessing is not thread-safe". So… That local from .http11 import SyncHTTP11Connection import might be the culprit — I didn't know it was a source of lack of thread-safety.

florimondmanca on 20 Nov 2020

🚀1

Yup — moving the local import of SyncHTTP11Connection to a global one (which is fine — we don't need a local import there, since h11 is always installed) fixes it on my side.

florimondmanca on 20 Nov 2020

🚀1

Was this page helpful?

0 / 5 - 0 ratings