Python: IncompleteRead while watching

Created on 26 Sep 2019 · 11Comments · Source: kubernetes-client/python

What happened (please include outputs or screenshots):

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 603, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 362, in _error_catcher
    yield
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 668, in read_chunked
    self._update_chunk_length()
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 607, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "controller.py", line 21, in <module>
    for event in stream:
  File "/usr/local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 134, in stream
    for line in iter_resp_lines(resp):
  File "/usr/local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 47, in iter_resp_lines
    for seg in resp.read_chunked(decode_content=False):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 696, in read_chunked
    self._original_response.close()
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 380, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

What you expected to happen:

The watcher to always recover and continue watching for events.

How to reproduce it (as minimally and precisely as possible):

Tricky. I've only seen it once and it was random. My application just watches a CRD for changes.

Environment:

Kubernetes version (kubectl version): v1.12.10-eks
Python version (python --version) 3.7.0
Python client version (pip list | grep kubernetes) 9.0.0

Duplicate issues:

Identical issues that were close by the bot: https://github.com/kubernetes-client/python/issues/728 https://github.com/kubernetes-client/python/issues/540

kinbug lifecyclrotten

Source

max-rocket-internet

All 11 comments

There should be a PR in progress: kubernetes-client/python-base#133

micw523 on 1 Oct 2019

🎉1

Hey @micw523 How to work around this? Seems the PR still be not merged. Thanks.

jinchihe on 19 Dec 2019

Hi @jinchihe , @roycaihw might have a better idea about this.

micw523 on 19 Dec 2019

@roycaihw is there way to work around the issue? except shorten waiting time. Thanks a lot!

jinchihe on 20 Dec 2019

I worked around this via @retrying.retry. :-) but still I think we need the enhancement, thanks.

jinchihe on 30 Dec 2019

I worked around this via @retrying.retry

How does that work?

max-rocket-internet on 2 Jan 2020

@max-rocket-internet I added the @retrying.retry(wait_fixed=1000, stop_max_attempt_number=20) for my watching function :-)

jinchihe on 3 Jan 2020

👍1

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 2 Apr 2020

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 2 May 2020

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 1 Jun 2020

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.