Docker-py: error while using docker for mac client 2.5.0.0

[mattmoskwa]

Using python 3.8.5 and docker-py version 4.3.1

  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/models/containers.py", line 887, in get
    resp = self.client.api.inspect_container(container_id)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/api/container.py", line 771, in inspect_container
    self._get(self._url("/containers/{0}/json", container)), True
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/api/client.py", line 228, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/sessions.py", line 685, in send
    r.content
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/models.py", line 754, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

[mconcas]

Hello, same/similar issue.

Python version: 3.8.6
Docker version: 2.5.0
docker-py version: 4.3.1
urllib3 version: 1.25.11

reproducer:

import docker

client = docker.from_env()
client.containers.get("<non-existent-container-name>")

expected result:
Exception at docker api level (docker.errors.NotFound or docker.errors.APIError)

result:
```python
Traceback (most recent call last):
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 696, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 436, in _error_catcher
yield
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 763, in read_chunked
self._update_chunk_length()
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 700, in _update_chunk_length
raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/models.py", line 751, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 571, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 792, in read_chunked
self._original_response.close()
File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 454, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "reproducer.py", line 5, in
client.containers.get("asdrubale")
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/models/containers.py", line 887, in get
resp = self.client.api.inspect_container(container_id)
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, args, *kwargs)
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/api/container.py", line 771, in inspect_container
self._get(self._url("/containers/{0}/json", container)), True
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, args, *kwargs)
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/api/client.py", line 228, in _get
return self.get(url, *self._set_request_timeout(kwargs))
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
return self.request('GET', url, *kwargs)
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/sessions.py", line 685, in send
r.content
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/models.py", line 829, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/models.py", line 754, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))```

[dkleissa]

An interesting observation on various version. I'm seeing this same issue on:

• latest Docker Desktop for Mac stable channel (2.5.0.0)
• latest Docker Desktop for Windows edge channel (2.4.2.0) on HyperV

I do not see this issue on:

• latest Docker Desktop for Windows stable channel (2.5.0.0) on WSL2
• latest Docker Desktop for Windows stable channel (2.5.0.0) on HyperV

[dkleissa]

We use this pattern to check if a container exists already before starting a container or when fetching the status of a container, as our software uses deterministic container names. In the case where the container isn't running we catch docker.errors.NotFound, which is now broken as @mconcas points out. After initial testing, it does appear this is the only place in the API that is broken like this.

A _very_ temporary fix is catching the unexpected error, again this is unexpected, fragile, and possibly not useful for everyone's use case:

import docker

client = docker.from_env()
try:
    client.containers.get("<non-existent-container-name>")
except docker.errors.NotFound:
    pass
except requests.exceptions.ChunkedEncodingError:
    pass

[reynoldsdj]

We use this pattern to check if a container exists already before starting a container or when fetching the status of a container, as our software uses deterministic container names. In the case where the container isn't running we catch docker.errors.NotFound, which is now broken as @mconcas points out. After initial testing, it does appear this is the only place in the API that is broken like this.

A _very_ temporary fix is catching the unexpected error, again this is unexpected, fragile, and possibly not useful for everyone's use case:

import docker

client = docker.from_env()
try:
    client.containers.get("<non-existent-container-name>")
except docker.errors.NotFound:
    pass
except requests.exceptions.ChunkedEncodingError:
    pass

I have the same exact problem, and I implemented the same fix. I just came here to open an issue. :)

Python 3.7.9 / docker-py 4.3.1 / Docker:

[tophers42]

Seeing the same issue using wsl1 on windows:

Python 3.8.2
Linux 5.4.39-linuxkit x86_64
Docker version 18.09.1, build 4c52b90
docker-py 4.3.1

[kevinkwoncareem]

Mac facing the same problem :) hope it gets resolved! thanks guys

[mconcas]

Also on Linux:

Server Version: 19.03.13

Python 3.6.8 (default, Apr  2 2020, 13:34:55)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
>>> client = docker.from_env()
>>> client.containers.get("<non-existent-container-name>")
Traceback (most recent call last):
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status
    response.raise_for_status()
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.40/containers/%3Cnon-existent-container-name%3E/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/models/containers.py", line 887, in get
    resp = self.client.api.inspect_container(container_id)
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/api/container.py", line 771, in inspect_container
    self._get(self._url("/containers/{0}/json", container)), True
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/api/client.py", line 265, in _result
    self._raise_for_status(response)
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("No such container: <non-existent-container-name>")

[avirshup]

Same issue - I was able to downgrade to 2.4 (link) to get it working again, no factory reset necessary.

[yannickperrenet]

They have just released version 2.5.0.1 which states to have fixed another Mac related issue (as you can see here in their docs). Not sure whether that also fixes this issue (which I sadly cannot test as I do not have a Mac), but I figured it was worth mentioning.

[reynoldsdj]

Docker 2.5.0.1 _does not_ fix this issue on the mac. I upgraded to 2.5.0.1 and tested it. 2.5.0.1 fixes an issue with Big Sur and Virtualbox.

[JukkaKeisala]

Will test it soon. Somehow new docker-desktop downgraded kubernetes to 1.16.6-beta.0 from 1.19.3 (by using kubectl get nodes), but that's not related to this issue.

[ch3ck]

This issue is affecting our container builds, I don't know if there's a release coming out soon to fix this.

[mconcas]

As a side note: the CI does not catch the problem because the default docker version in Travis is still 18.x ...
This will be true even if they update from xenial to bionic and will require further action on the CI config.
Moreover I could not find the unit test that checks for this.

[thaJeztah]

I think this may be the same issue as https://github.com/docker/for-mac/issues/5025 / https://github.com/docker/for-mac/issues/5058 / https://github.com/docker/for-mac/issues/5059. More details can be found in this comment: https://github.com/docker/for-mac/issues/5059#issuecomment-726036933

[thaJeztah]

If the python code you're running is running on the mac itself, I'm not sure if there's a workaround, but if the code is running inside a container, you can (as a workaround) bind-mount /var/run/docker.sock.raw instead of /var/run/docker.sock to skip the docker desktop API proxy

[thaJeztah]

Oh, actually, there might be a workaround; the "raw" socket is also available on the mac side in;

$ ls -la ~/Library/Containers/com.docker.docker/Data/docker.raw.sock

srwxr-xr-x   1 sebastiaan  staff         0 Nov 16 11:24 docker.raw.sock=

e.g.;

$ docker -H unix:///Users/sebastiaan/Library/Containers/com.docker.docker/Data/docker.raw.sock version
Client: Docker Engine - Community
 Version:           20.10.0-beta1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        ac365d7
 Built:             Tue Oct 13 18:13:53 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true
...

⚠️ ⚠️ BIG FAT WARNING!!! ⚠️ ⚠️

I think the name of that socket will be renamed soon, so only use this as a temporary workaround.
Also, when "skipping" the API proxy, some things may not work as expected (for example, paths
used in bind-mounts are not rewritten, network settings may not be adapted for how Docker Desktop is setup)

☝️ ☝️ ☝️ ☝️ IOW: the recommended socket to use will be /var/run/docker.sock, which is the official API to connect (for reasons above).

[reynoldsdj]

I think this may be the same issue as docker/for-mac#5025 / docker/for-mac#5058 / docker/for-mac#5059. More details can be found in this comment: docker/for-mac#5059 (comment)

Yes, I believe that it the exact cause of this issue. https://github.com/docker/for-mac/issues/5025

[thaJeztah]

looks like the issue should be fixed in the latest edge release that was just published https://github.com/docker/for-mac/issues/5025#issuecomment-729826423

[kilrau]

Latest edge doesn't fix this for me. Can someone confirm?

[reynoldsdj]

I concur. I just upgraded, and I am seeing the same error.

[reynoldsdj]

Error from trying to find a non-existent container (test suite log):

2020-11-20 08:42:56,620 - tests.fixtures.db - WARNING - Received ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) due to docker bug

[thaJeztah]

Can confirm that I still see the problem;

Incorrect response with the proxied socket;

$ curl -v --unix-socket /var/run/docker.sock http://localhost/containers/nosuchcontainer/json

*   Trying /var/run/docker.sock:0...
* Connected to localhost (docker.sock) port 80 (#0)
> GET /containers/nosuchcontainer/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Api-Version: 1.41
< Content-Type: application/json
< Date: Fri, 20 Nov 2020 14:23:21 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/20.10.0-rc1 (linux)
< Transfer-Encoding: chunked
<
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining

Works correct with the 'raw' socket:

$ curl -v --unix-socket ~/Library/Containers/com.docker.docker/Data/docker.raw.sock http://localhost/containers/nosuchcontainer/json

*   Trying /Users/sebastiaan/Library/Containers/com.dock:0...
* Connected to localhost (/Users/sebastiaan/Library/Containers/com.dock) port 80 (#0)
> GET /containers/nosuchcontainer/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Api-Version: 1.41
< Content-Type: application/json
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/20.10.0-rc1 (linux)
< Date: Fri, 20 Nov 2020 14:22:40 GMT
< Content-Length: 49
<
{"message":"No such container: nosuchcontainer"}
* Connection #0 to host localhost left intact

[thaJeztah]

This should be fixed in docker desktop 2.5.2 and up;

curl -v --unix-socket /var/run/docker.sock http://localhost/containers/nosuchcontainer/json

*   Trying /var/run/docker.sock:0...
* Connected to localhost (docker.sock) port 80 (#0)
> GET /containers/nosuchcontainer/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Api-Version: 1.41
< Content-Type: application/json
< Date: Tue, 01 Dec 2020 16:22:01 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/20.10.0-rc1 (linux)
< Transfer-Encoding: chunked
<
{"message":"No such container: nosuchcontainer"}
* Connection #0 to host localhost left intact

[DmytroSokhach]

@thaJeztah I'm trying to migrate from 2.3.2 to higher, we are uing docker stack like:

volumes:
   /var/run/docker.sock.raw:/var/run/docker.sock

Is there a way not to add ".raw", cause it is not backward compatible and will fail for those who stay on 2.3.2 (we have way to many non programmer users who will not know what to do with this)