Poetry: Frequent intermittent connection failures
- [x] I am on the latest Poetry version.
- [x] I have searched the issues of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).
- OS version and name: Debian GNU/Linux 10 (buster)
- Poetry version: 1.1.3
- Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/maroux/5bfc38375acd0e1c1a710fc0e62b8e93
Issue
Environment: Docker debian slim buster image running in Google Cloud Platform Cloud Build
Installation: poetry install -vvv --no-root --no-dev
Outcome: Fails with networking error intermittently, but frequently enough that upgrade to poetry v1.1 is blocked on solving this issue.
Notes: Older version (v1.0.9) works fine but is slow because of old installer.
Verbose logs:
16:05:17 Step #2 - "build": Skipping virtualenv creation, as specified in config file.
16:05:17 Step #2 - "build": Installing dependencies from lock file
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": Finding the necessary packages for the current system
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": Package operations: 17 installs, 1 update, 0 removals, 11 skipped
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": ? Installing certifi (2020.6.20): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Installing chardet (3.0.4): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Installing idna (2.10): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Installing markupsafe (1.1.1)
16:05:17 Step #2 - "build": ? Installing pyparsing (2.4.7): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Installing pytz (2020.1): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Installing six (1.15.0): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Installing urllib3 (1.25.10): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Installing alabaster (0.7.12)
16:05:17 Step #2 - "build": ? Installing babel (2.8.0)
16:05:17 Step #2 - "build": ? Updating colorama (0.4.4 -> 0.4.3): Skipped for the following reason: Not needed for the current environment
16:05:17 Step #2 - "build": ? Installing docutils (0.16): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Installing imagesize (1.2.0)
16:05:17 Step #2 - "build": ? Installing jinja2 (2.11.2)
16:05:17 Step #2 - "build": ? Installing packaging (20.4): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Updating pygments (2.7.1 -> 2.6.1)
16:05:17 Step #2 - "build": ? Installing requests (2.24.0): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build": ? Installing snowballstemmer (2.0.0)
16:05:17 Step #2 - "build": ? Installing sphinxcontrib-applehelp (1.0.1)
16:05:17 Step #2 - "build": ? Installing sphinxcontrib-devhelp (1.0.1)
16:05:17 Step #2 - "build": ? Installing sphinxcontrib-htmlhelp (1.0.2)
16:05:17 Step #2 - "build": ? Installing sphinxcontrib-jsmath (1.0.1)
16:05:17 Step #2 - "build": ? Installing sphinxcontrib-qthelp (1.0.2)
16:05:17 Step #2 - "build": ? Installing sphinxcontrib-serializinghtml (1.1.3)
16:05:17 Step #2 - "build": ? Installing commonmark (0.9.1)
16:05:17 Step #2 - "build": ? Installing sphinx (3.1.2)
16:05:17 Step #2 - "build": Retrying HTTP request in 0.5 seconds.
16:05:17 Step #2 - "build": ? Installing recommonmark (0.5.0)
16:05:17 Step #2 - "build": ? Installing sphinx-autodoc-typehints (1.11.0)
16:05:17 Step #2 - "build": ? Installing sphinx-rtd-theme (0.4.3)
16:05:17 Step #2 - "build": Retrying HTTP request in 0.5 seconds.
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": Stack trace:
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 13 ~/.poetry/lib/poetry/installation/executor.py:199 in _execute_operation
16:05:17 Step #2 - "build": 197?
16:05:17 Step #2 - "build": 198? try:
16:05:17 Step #2 - "build": ? 199? result = self._do_execute_operation(operation)
16:05:17 Step #2 - "build": 200? except EnvCommandError as e:
16:05:17 Step #2 - "build": 201? if e.e.returncode == -2:
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 12 ~/.poetry/lib/poetry/installation/executor.py:273 in _do_execute_operation
16:05:17 Step #2 - "build": 271? return 0
16:05:17 Step #2 - "build": 272?
16:05:17 Step #2 - "build": ? 273? result = getattr(self, "_execute_{}".format(method))(operation)
16:05:17 Step #2 - "build": 274?
16:05:17 Step #2 - "build": 275? if result != 0:
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 11 ~/.poetry/lib/poetry/installation/executor.py:408 in _execute_install
16:05:17 Step #2 - "build": 406?
16:05:17 Step #2 - "build": 407? def _execute_install(self, operation): # type: (Install) -> None
16:05:17 Step #2 - "build": ? 408? return self._install(operation)
16:05:17 Step #2 - "build": 409?
16:05:17 Step #2 - "build": 410? def _execute_update(self, operation): # type: (Update) -> None
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 10 ~/.poetry/lib/poetry/installation/executor.py:434 in _install
16:05:17 Step #2 - "build": 432? archive = self._download_link(operation, Link(package.source_url))
16:05:17 Step #2 - "build": 433? else:
16:05:17 Step #2 - "build": ? 434? archive = self._download(operation)
16:05:17 Step #2 - "build": 435?
16:05:17 Step #2 - "build": 436? operation_message = self.get_operation_message(operation)
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 9 ~/.poetry/lib/poetry/installation/executor.py:575 in _download
16:05:17 Step #2 - "build": 573?
16:05:17 Step #2 - "build": 574? def _download(self, operation): # type: (Operation) -> Path
16:05:17 Step #2 - "build": ? 575? link = self._chooser.choose_for(operation.package)
16:05:17 Step #2 - "build": 576?
16:05:17 Step #2 - "build": 577? return self._download_link(operation, link)
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 8 ~/.poetry/lib/poetry/installation/chooser.py:60 in choose_for
16:05:17 Step #2 - "build": 58? """
16:05:17 Step #2 - "build": 59? links = []
16:05:17 Step #2 - "build": ? 60? for link in self._get_links(package):
16:05:17 Step #2 - "build": 61? if link.is_wheel and not Wheel(link.filename).is_supported_by_environment(
16:05:17 Step #2 - "build": 62? self._env
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 7 ~/.poetry/lib/poetry/installation/chooser.py:94 in _get_links
16:05:17 Step #2 - "build": 92? repository = self._pool.repository(package.source_reference)
16:05:17 Step #2 - "build": 93?
16:05:17 Step #2 - "build": ? 94? links = repository.find_links_for_package(package)
16:05:17 Step #2 - "build": 95?
16:05:17 Step #2 - "build": 96? hashes = [f["hash"] for f in package.files]
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 6 ~/.poetry/lib/poetry/repositories/pypi_repository.py:245 in find_links_for_package
16:05:17 Step #2 - "build": 243?
16:05:17 Step #2 - "build": 244? def find_links_for_package(self, package):
16:05:17 Step #2 - "build": ? 245? json_data = self._get("pypi/{}/{}/json".format(package.name, package.version))
16:05:17 Step #2 - "build": 246? if json_data is None:
16:05:17 Step #2 - "build": 247? return []
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 5 ~/.poetry/lib/poetry/repositories/pypi_repository.py:319 in _get
16:05:17 Step #2 - "build": 317? def _get(self, endpoint): # type: (str) -> Union[dict, None]
16:05:17 Step #2 - "build": 318? try:
16:05:17 Step #2 - "build": ? 319? json_response = self.session.get(self._base_url + endpoint)
16:05:17 Step #2 - "build": 320? except requests.exceptions.TooManyRedirects:
16:05:17 Step #2 - "build": 321? # Cache control redirect loop.
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 4 ~/.poetry/lib/poetry/_vendor/py3.7/requests/sessions.py:543 in get
16:05:17 Step #2 - "build": 541?
16:05:17 Step #2 - "build": 542? kwargs.setdefault('allow_redirects', True)
16:05:17 Step #2 - "build": ? 543? return self.request('GET', url, **kwargs)
16:05:17 Step #2 - "build": 544?
16:05:17 Step #2 - "build": 545? def options(self, url, **kwargs):
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 3 ~/.poetry/lib/poetry/_vendor/py3.7/requests/sessions.py:530 in request
16:05:17 Step #2 - "build": 528? }
16:05:17 Step #2 - "build": 529? send_kwargs.update(settings)
16:05:17 Step #2 - "build": ? 530? resp = self.send(prep, **send_kwargs)
16:05:17 Step #2 - "build": 531?
16:05:17 Step #2 - "build": 532? return resp
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 2 ~/.poetry/lib/poetry/_vendor/py3.7/requests/sessions.py:643 in send
16:05:17 Step #2 - "build": 641?
16:05:17 Step #2 - "build": 642? # Send the request
16:05:17 Step #2 - "build": ? 643? r = adapter.send(request, **kwargs)
16:05:17 Step #2 - "build": 644?
16:05:17 Step #2 - "build": 645? # Total elapsed time of the request (approximately)
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": 1 ~/.poetry/lib/poetry/_vendor/py3.7/cachecontrol/adapter.py:53 in send
16:05:17 Step #2 - "build": 51? request.headers.update(self.controller.conditional_headers(request))
16:05:17 Step #2 - "build": 52?
16:05:17 Step #2 - "build": ? 53? resp = super(CacheControlAdapter, self).send(request, **kw)
16:05:17 Step #2 - "build": 54?
16:05:17 Step #2 - "build": 55? return resp
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": ConnectionError
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
16:05:17 Step #2 - "build":
16:05:17 Step #2 - "build": at ~/.poetry/lib/poetry/_vendor/py3.7/requests/adapters.py:498 in send
16:05:17 Step #2 - "build": 494? low_conn.close()
16:05:17 Step #2 - "build": 495? raise
16:05:17 Step #2 - "build": 496?
16:05:17 Step #2 - "build": 497? except (ProtocolError, socket.error) as err:
16:05:17 Step #2 - "build": ? 498? raise ConnectionError(err, request=request)
16:05:17 Step #2 - "build": 499?
16:05:17 Step #2 - "build": 500? except MaxRetryError as e:
16:05:17 Step #2 - "build": 501? if isinstance(e.reason, ConnectTimeoutError):
16:05:17 Step #2 - "build": 502? # TODO: Remove this in 3.0.0: see #2811
16:05:17 Step #2 - "build":
maroux
All 19 comments
I can confirm the same issue with Poetry 1.1.2. Every other install fails with a ConnectionError.
winpat
on 16 Oct 2020
Hmm, we saw this intermittently in our own CI a while ago, but was restricted to Cirrus CI FreeBSD instances (which runs on Google Cloud).
We might be able to do a bug fix to try recreate the session, but I suspect the issue might be at the process/scoket level.
@winpat what environment are you seeing your errors in?
I am really inclined to move http logic to httpx in the next minor to handle concurrent scenarios a bit better.
abn
on 16 Oct 2020
@winpat what environment are you seeing your errors in?
I observed the problem in various GitLab CI pipelines (GitLab.com, not self hosted). If I remember correctly then GitLab also uses GCP.
winpat
on 16 Oct 2020
We are having the same issue with our CircleCI builds (poetry 1.1.2, VM with ubuntu-1604:202004-01 image and python 3.7)
...
#21 423.2
#21 423.2 ConnectionError
#21 423.2
#21 423.2 ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
#21 423.2
#21 423.2 at ~/.poetry/lib/poetry/_vendor/py3.7/requests/adapters.py:498 in send
#21 423.4 494│ low_conn.close()
#21 423.4 495│ raise
#21 423.4 496│
#21 423.4 497│ except (ProtocolError, socket.error) as err:
#21 423.4 → 498│ raise ConnectionError(err, request=request)
#21 423.4 499│
#21 423.4 500│ except MaxRetryError as e:
#21 423.4 501│ if isinstance(e.reason, ConnectTimeoutError):
#21 423.4 502│ # TODO: Remove this in 3.0.0: see #2811
#21 423.4
#21 426.5 • Installing pytest-timeout (1.4.2)
#21 427.7 • Installing scipy (1.5.2)
#21 428.1 • Installing threadpoolctl (2.1.0)
#21 ERROR: executor failed running [/bin/sh -c $POETRY_BINARY install]: runc did not terminate sucessfully
dariobig
on 17 Oct 2020
I am not entirely sure if this will fix the issue, since I am assuming the real issue here is some sort of networking layer throttling being hit. If some of you can please try the fix at #3230, it would be helpful to know if session reuse across thread exeuctions was the issue here.
Using pipx
pipx install --suffix=@3230 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/3230/head'
Using a container (podman | docker)
podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/3230/head
poetry new foobar
pushd foobar
sed -i /pytest/d pyproject.toml
poetry add pycowsay
# commands to reproduce your build
EOF
@maroux @winpat @dariobig can you please verify if the fix above resolves the issue for your environments. If it does I can get that into the next patch.
abn
on 19 Oct 2020
@abn Thanks for your help! I've added your fix to a couple of different projects. It will take a day or two until I can whether it's working or not.
winpat
on 20 Oct 2020
@maroux @winpat @dariobig can you please verify if the fix above resolves the issue for your environments. If it does I can get that into the next patch.
I've run it four times now and it's worked every time. I believe the PR does fix this issue.
maroux
on 20 Oct 2020
Unfortunately I can't test it in our environment, I've turned off "new
installer" instead and that fixed the issue for us.
On Tue, Oct 20, 2020 at 11:37 AM Aniruddha Maru notifications@github.com
wrote:
@maroux https://github.com/maroux @winpat https://github.com/winpat
@dariobig https://github.com/dariobig can you please verify if the fix
above resolves the issue for your environments. If it does I can get that
into the next patch.I've run it four times now and it's worked every time. I believe the PR
does fix this issue.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/python-poetry/poetry/issues/3219#issuecomment-712942233,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABFE553C5KZ7MJQTQMHVTSLWVEPANCNFSM4SSUHTOA
.
dariobig
on 20 Oct 2020
@maroux @winpat @dariobig can you please verify if the fix above resolves the issue for your environments. If it does I can get that into the next patch.
I've run it four times now and it's worked every time. I believe the PR does fix this issue.
I also ran it a couple of times without any issues so far 👍.
winpat
on 20 Oct 2020
I am actually seeing a similar issue during test set up which only occurs for Python 3.6 with FreeBSD for the poetry PR #3259.
12 .tox/py/lib/python3.6/site-packages/poetry/installation/executor.py:273 in _do_execute_operation
result = getattr(self, "_execute_{}".format(method))(operation)
11 .tox/py/lib/python3.6/site-packages/poetry/installation/executor.py:408 in _execute_install
return self._install(operation)
10 .tox/py/lib/python3.6/site-packages/poetry/installation/executor.py:434 in _install
archive = self._download(operation)
9 .tox/py/lib/python3.6/site-packages/poetry/installation/executor.py:575 in _download
link = self._chooser.choose_for(operation.package)
8 .tox/py/lib/python3.6/site-packages/poetry/installation/chooser.py:60 in choose_for
for link in self._get_links(package):
7 .tox/py/lib/python3.6/site-packages/poetry/installation/chooser.py:94 in _get_links
links = repository.find_links_for_package(package)
6 .tox/py/lib/python3.6/site-packages/poetry/repositories/pypi_repository.py:245 in find_links_for_package
json_data = self._get("pypi/{}/{}/json".format(package.name, package.version))
5 .tox/py/lib/python3.6/site-packages/poetry/repositories/pypi_repository.py:319 in _get
json_response = self.session.get(self._base_url + endpoint)
4 .tox/py/lib/python3.6/site-packages/requests/sessions.py:543 in get
return self.request('GET', url, **kwargs)
3 .tox/py/lib/python3.6/site-packages/requests/sessions.py:530 in request
resp = self.send(prep, **send_kwargs)
2 .tox/py/lib/python3.6/site-packages/requests/sessions.py:643 in send
r = adapter.send(request, **kwargs)
1 .tox/py/lib/python3.6/site-packages/cachecontrol/adapter.py:53 in send
resp = super(CacheControlAdapter, self).send(request, **kw)
ConnectionError
('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
at .tox/py/lib/python3.6/site-packages/requests/adapters.py:498 in send
494│ low_conn.close()
495│ raise
496│
497│ except (ProtocolError, socket.error) as err:
→ 498│ raise ConnectionError(err, request=request)
499│
500│ except MaxRetryError as e:
501│ if isinstance(e.reason, ConnectTimeoutError):
502│ # TODO: Remove this in 3.0.0: see #2811
ERROR: InvocationError for command /tmp/cirrus-ci-build/.tox/py/bin/poetry install -vv --no-root (exited with code 1)
___________________________________ summary ____________________________________
ERROR: py: commands failed
DustinMoriarty
on 22 Oct 2020
~when is this gonna be fixed? this morning: again 9 consecutive travis builds before i got thru my poetry install step. poetry version: 1.1.2~
i've now recognized my posts here and on the other issues were unnecesarily blunt and had a tone i dont stand for. So i apologize for my behavior and wish no harm was done. I was under stress and this was an accidental side-effect.
Hope you all have a nice day and i will see if i can find a way to contribute constructively to this great project going forward.
...johannes
sloev
on 22 Oct 2020
@sloev as I mentioned on the pull request (https://github.com/python-poetry/poetry/pull/3230#issuecomment-713551777), this is an open source community project, not a product you have a paid support for. This is maintained by a small group of volunteers who spend their spare time on this. You demanding issues be fixed and spamming issue threads is not going to make anything go any faster. While we can appreciate that this is causing issues with your workflow, we have mentioned in the release post how to disable the experimental installer to unblock yourself. Please keep the conversation constructive and civil.
abn
on 22 Oct 2020
@DustinMoriarty yeah; we noticed that. This should go away once 1.1.4 is out. Hopefully the fix in #3230 is sufficient.
abn
on 22 Oct 2020
This was resolved with 1.1.4.
abn
on 24 Oct 2020
uh-oh just saw this now on 1.1.4:
13:08:47 Step #2 - "build": ChunkedEncodingError
13:08:47 Step #2 - "build":
13:08:47 Step #2 - "build": ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
13:08:47 Step #2 - "build":
13:08:47 Step #2 - "build": at ~/.poetry/lib/poetry/_vendor/py3.7/requests/models.py:754 in generate
13:08:47 Step #2 - "build": 750? try:
13:08:47 Step #2 - "build": 751? for chunk in self.raw.stream(chunk_size, decode_content=True):
13:08:47 Step #2 - "build": 752? yield chunk
13:08:47 Step #2 - "build": 753? except ProtocolError as e:
13:08:47 Step #2 - "build": ? 754? raise ChunkedEncodingError(e)
13:08:47 Step #2 - "build": 755? except DecodeError as e:
13:08:47 Step #2 - "build": 756? raise ContentDecodingError(e)
13:08:47 Step #2 - "build": 757? except ReadTimeoutError as e:
13:08:47 Step #2 - "build": 758? raise ConnectionError(e)
13:08:47 Step #2 - "build":
13:08:47 Step #2 - "build": ? Installing snowballstemmer (2.0.0)
13:08:47 Step #2 - "build": ? Installing sphinxcontrib-applehelp (1.0.1)
13:09:01 ERROR: (gcloud.builds.submit) build 17e8547a-8452-4010-a146-81c7f21a4309 completed with status "FAILURE"
might be unrelated?
maroux
on 27 Oct 2020
@maroux I suspect this might be different as this is connection being broken during a download.
abn
on 27 Oct 2020
We're still seeing exactly this problem on our CI builds (azure pipelines) with poetry-1.1.4, our docker base image is python:3.6-slim-buster just like OP:
#18 13.66 • Installing black (19.10b0)
#18 13.83
#18 13.83 ConnectionError
#18 13.83
#18 13.83 ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
#18 13.83
#18 13.83 at /usr/local/lib/python3.6/site-packages/requests/adapters.py:498 in send
#18 13.87 494│ low_conn.close()
#18 13.87 495│ raise
#18 13.87 496│
#18 13.87 497│ except (ProtocolError, socket.error) as err:
#18 13.87 → 498│ raise ConnectionError(err, request=request)
#18 13.87 499│
#18 13.87 500│ except MaxRetryError as e:
#18 13.87 501│ if isinstance(e.reason, ConnectTimeoutError):
#18 13.87 502│ # TODO: Remove this in 3.0.0: see #2811
Since there's a pretty big list of packages to install, this happens almost every build. We're pinning to poetry < 1.1 for now as a workaround.
Korijn
on 5 Nov 2020
Since there's a pretty big list of packages to install, this happens almost every build. We're pinning to poetry < 1.1 for now as a workaround.
As per https://python-poetry.org/blog/announcing-poetry-1-1-0.html#brand-new-installer adding something like the following to your CI builds might work:
SimonBiggs
on 22 Dec 2020
Related issues
sobolevn
·
3Comments
EdgyEdgemond
·
3Comments
probablykasper
·
3Comments
jbarlow83
·
3Comments
thmo
·
3Comments
Most helpful comment
This was resolved with
1.1.4.